Prototype-Based Image Prompting for Weakly Supervised Histopathological Image Segmentation

Qingchen Tang, Lei Fan, Maurice Pagnucco, Yang Song

Introduction

Weakly Supervised Segmentation (WSS) aims to train models using only image-level labels, significantly reducing the annotation burden in medical imaging. Traditional methods like Class Activation Maps (CAMs) often produce incomplete masks because they only focus on the most discriminative regions. While vision-language models (e.g., CLIP) have introduced text-based prompting to guide segmentation, textual descriptions are often too vague for the high-stakes visual nuances of histopathology.

The Challenge: Homogeneity and Heterogeneity

Histopathological images present two primary challenges that text prompts fail to address effectively:

  1. Inter-class Homogeneity: Different tissue types (e.g., tumor vs. lymphocytic) can look remarkably similar in texture and color.
  2. Intra-class Heterogeneity: The same tissue type (e.g., tumor) can exhibit vast morphological differences across different patients due to varying staining techniques or cell shapes.
Comparison of supervision methods and the visual challenges of inter-class homogeneity and intra-class heterogeneity in histopathology.

Proposed Method: PBIP

The Prototype-Based Image Prompting (PBIP) framework replaces blunt text prompts with a bank of visual "prototypes" to guide the model more precisely.

1. Image Bank Construction

Instead of using words, the researchers cluster training images using K-Means to find representative "prototypes" for each class. This ensures the model has a visual reference for the many different ways a tissue type might appear, directly addressing intra-class heterogeneity.

2. Architecture and Refinement

The PBIP architecture consists of several integrated modules:

  • Classification Network: Uses a SegFormer backbone to compare features from a query image against the stored prototypes to generate an initial "pseudo-mask" (M).
  • Feature Matching Network: Refines these masks by separating predicted foreground regions from the background. It uses a similarity matching loss ($L_{SIM}$) to ensure identified features align closely with the correct prototypes in the bank.
  • MedCLIP Integration: The system leverages MedCLIP, a version of CLIP pre-trained specifically on medical images, ensuring the underlying feature extraction is tuned for biology.
Detailed overview of the PBIP framework, illustrating the Image Bank clustering, Feature Matching Network, and Classification pipeline.

Key Results and Insights

PBIP set new state-of-the-art benchmarks for weakly supervised segmentation in histopathology across datasets like BCSS-WSSS and LUAD-HistoSeg.

  • Performance Boost: PBIP outperformed the previous best method (Proto2Seg) by over 1.5% in mean Intersection over Union (mIoU).
  • Zero-Shot Superiority: In zero-shot classification settings, using image prototypes led to an F1 score of 0.874, dwarfing the 0.310 achieved by text-based prompts using the same underlying model.
  • Qualitative Improvement: Visualizations show that PBIP produces much "cleaner" and more complete masks compared to previous methods, which often suffered from noisy activations in background regions.

Conclusion

By shifting from text-based to image-based prompting, PBIP effectively addresses the inherent visual ambiguities of histopathology. While the framework's performance is tied to the quality of the pre-trained encoder and requires careful hyperparameter balancing, it represents a significant advancement in making weakly supervised medical image analysis more robust and accurate.

Summary updated

Abstract

Weakly supervised image segmentation with image-level labels has drawn attention due to the high cost of pixel-level annotations. Traditional methods using Class Activation Maps (CAMs) often highlight only the most discriminative regions, leading to incomplete masks. Recent approaches that introduce textual information struggle with histopathological images due to inter-class homogeneity and intra-class heterogeneity. In this paper, we propose a prototype-based image prompting framework for histopathological image segmentation. It constructs an image bank from the training set using clustering, extracting multiple prototype features per class to capture intra-class heterogeneity. By designing a matching loss between input features and class-specific prototypes using contrastive learning, our method addresses inter-class homogeneity and guides the model to generate more accurate CAMs. Experiments on four datasets (LUAD-HistoSeg, BCSS-WSSS, GCSS, and BCSS) show that our method outperforms existing weakly supervised segmentation approaches, setting new benchmarks in histopathological image segmentation.
BibTeX
@inproceedings{tang2025prototype,
  author = {Qingchen Tang and Lei Fan and Maurice Pagnucco and Yang Song},
  title = {{Prototype-Based Image Prompting for Weakly Supervised Histopathological Image Segmentation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02818},
}
Figure