TL;DR
This paper introduces TOP (Two-level Prompt Learning), a novel framework that tackles the challenging task of classifying gigapixel pathology images with very few labeled examples. By leveraging the medical knowledge of GPT-4 to guide a vision-language model (CLIP), TOP achieves state-of-the-art classification at both the whole-slide and individual-patch levels using as few as one to sixteen training samples.
What Problem Does This Paper Address?
In digital pathology, doctors work with Whole Slide Images (WSIs)—gigapixel-resolution scans of tissue samples. Because these images are too massive to feed directly into deep learning models, researchers typically chop them into thousands of smaller patches.
Standard practice treats WSI classification as a Multiple Instance Learning (MIL) problem: the entire slide is a "bag," and the patches are "instances." Usually, only the slide has a label (e.g., "cancerous"), while the individual patches remain unlabeled.
While MIL works well with massive datasets, clinical reality often presents a data scarcity problem due to rare diseases or privacy concerns. The authors define a new, highly challenging task: Few-shot Weakly-supervised WSI Classification (FSWC). The goal is to accurately classify both the entire slide and the individual patches using only a handful of labeled slides (e.g., 1, 2, 4, 8, or 16 slides per class) and zero patch-level labels.
Methods
To solve the FSWC problem, the authors propose TOP (TwO-level Prompt Learning), which integrates the visual feature extraction of CLIP with the rich medical knowledge of GPT-4.
The framework operates on two distinct levels:
- Instance-Level (Patch) Guidance: Instead of using standard mathematical pooling (like max or mean pooling) to combine patch features into a single slide feature, TOP introduces prompt-guided pooling. The authors ask GPT-4 to describe the visual characteristics of various tissue phenotypes (e.g., squamous epithelium, lymphocytes). These descriptions form "instance prototypes." The model calculates the similarity between each patch and these prototypes, using the scores to intelligently weight and aggregate the patches into a single "bag feature."
- Bag-Level (Slide) Classification: At the slide level, GPT-4 is queried to provide detailed visual descriptions of specific diseases (e.g., Lung Adenocarcinoma vs. Lung Squamous Cell Carcinoma). These descriptions are combined with learnable prompt tokens (similar to CoOp) to form a "bag-level prompt group." The aggregated bag feature is then matched against these prompts to make the final slide classification.
To prevent the learned instance prototypes from blending together and becoming identical during training, the authors introduce a unique auxiliary diversity loss that forces the prototypes to remain distinct.
Datasets
The authors evaluated their method on three real-world WSI datasets representing different cancer types and medical centers:
- Camelyon 16: A breast cancer dataset containing true labels for both slides and individual patches (used to verify patch-level accuracy).
- TCGA-Lung Cancer: A public dataset for differentiating lung cancer subtypes.
- Cervical Cancer Dataset: An in-house dataset used to evaluate the model on challenging cervical cancer classifications.
Key Insights and Ideas
- LLMs as Clinical Priors: Instead of relying purely on data-driven features, TOP uses GPT-4 as an automated, highly detailed medical textbook. This injects expert-level domain knowledge into the vision model without requiring tedious manual prompt engineering by human pathologists.
- Two-Level Alignment is Crucial: Simply applying natural-image few-shot techniques (like CoOp) directly to the slide level yields poor results. Aligning language prompts at both the patch level (for aggregation) and the slide level (for classification) is what unlocks strong few-shot performance.
- Prototype Diversity: The inclusion of an auxiliary loss to keep instance prototypes distinct ensures that the model captures a wide variety of tissue structures across the slide, rather than hyper-focusing on just one visual feature.
Key Results
The TOP framework consistently outperformed all baseline methods (including Linear-Probe and standard CoOp) across all three datasets, often by a wide margin.
- On Camelyon 16 (Breast Cancer): In a 16-shot setting, TOP achieved a slide-level Area Under the Curve (AUC) of 0.8301 compared to CoOp's 0.8012. More impressively, its patch-level classification AUC reached 0.8896, significantly outperforming other methods at identifying exact cancerous regions.
- On TCGA-Lung Cancer: TOP achieved a 16-shot slide AUC of 0.8235 (outperforming CoOp at 0.7840) and maintained a high AUC of 0.7123 even in the extreme 1-shot scenario.
- On Cervical Cancer: On this highly challenging dataset, TOP demonstrated an average improvement of 6.3% over the second-best method across all shot configurations.
Limitations and Future Work
The primary limitation of TOP is its dependency on the quality of the large language model. If the LLM generates inaccurate or vague visual descriptions for a specific rare disease, the quality of both the prompt-guided pooling and the final classification will degrade.
Moving forward, the authors suggest exploring the integration of TOP with emerging, domain-specific pathology foundation models rather than general-domain vision-language models like standard CLIP.
Summary updated