Abstract
Weakly supervised image segmentation with image-level labels has drawn attention due to the high cost of pixel-level annotations. Traditional methods using Class Activation Maps (CAMs) often highlight only the most discriminative regions, leading to incomplete masks. Recent approaches that introduce textual information struggle with histopathological images due to inter-class homogeneity and intra-class heterogeneity. In this paper, we propose a prototype-based image prompting framework for histopathological image segmentation. It constructs an image bank from the training set using clustering, extracting multiple prototype features per class to capture intra-class heterogeneity. By designing a matching loss between input features and class-specific prototypes using contrastive learning, our method addresses inter-class homogeneity and guides the model to generate more accurate CAMs. Experiments on four datasets (LUAD-HistoSeg, BCSS-WSSS, GCSS, and BCSS) show that our method outperforms existing weakly supervised segmentation approaches, setting new benchmarks in histopathological image segmentation.
Introduction
In the field of digital pathology, the gold standard for training segmentation models is pixel-level annotation. However, generating these precise masks is a labor-intensive process requiring significant expertise from pathologists. Weakly Supervised Segmentation (WSS) offers a promising alternative by utilizing image-level labels—simple tags indicating the presence of a tissue type—to train models.
Traditional WSS methods typically rely on Class Activation Maps (CAMs). While effective at identifying discriminative regions, CAMs often fail to capture the full extent of a biological structure, focusing only on the most salient features. Recent attempts to bridge this gap using Vision-Language Models (VLMs) like CLIP have introduced text-based prompting. However, these methods struggle with the unique challenges of histopathology: inter-class homogeneity (different tissues looking similar) and intra-class heterogeneity (the same tissue type appearing differently across samples). This paper introduces the Prototype-Based Image Prompting (PBIP) framework to overcome these hurdles.
The PBIP Framework
PBIP shifts the focus from vague natural language descriptions to visual "prototypes" extracted directly from the training data. The framework consists of three primary components designed to navigate the visual complexity of medical imaging.
1. The Image Bank and Prototype Extraction
The core of PBIP is an image bank constructed from the training set. For each class, the framework applies K-Means clustering to the features of training images. This process identifies $K$ representative visual styles—or prototypes—per class. By maintaining multiple prototypes, the model can account for the diverse appearances a single tissue type might take due to varying staining techniques or pathological manifestations (intra-class heterogeneity).
2. ClassNet and Initial Segmentation
The framework employs a SegFormer backbone, referred to as ClassNet, to generate initial pseudo-masks. Unlike standard models that classify pixels in isolation, ClassNet compares the features of a query image against the prototypes stored in the image bank. This comparison provides a more robust foundation for identifying tissue boundaries than text-based priors.
3. ImgMatchNet and Contrastive Matching Loss
To refine the initial masks, the authors introduce ImgMatchNet. This component utilizes a contrastive matching loss to align the features of the predicted foreground regions with their corresponding class prototypes. Simultaneously, it ensures these features remain distinct from prototypes of other classes. This dual-action loss function is critical for resolving inter-class homogeneity, forcing the model to distinguish between subtly different tissue types.
4. Leveraging MedCLIP
PBIP integrates MedCLIP, a vision-language model pre-trained specifically on medical data. By using MedCLIP as the feature encoder, the framework benefits from domain-specific representations that are far more sensitive to pathological nuances than standard models trained on everyday photographs.
Experimental Evaluation
The PBIP framework was evaluated across four diverse H&E-stained histopathology datasets: BCSS-WSSS, LUAD-HistoSeg, GCSS, and BCSS.
Performance Benchmarks
PBIP consistently established new state-of-the-art (SOTA) results. On the BCSS-WSSS dataset, the model achieved a Mean Intersection over Union (mIoU) of 69.54%, representing a 1.68% improvement over the previous leading method, Proto2Seg. Similar gains were observed on LUAD-HistoSeg (76.44% mIoU) and GCSS (55.10% mIoU).
Visual vs. Textual Prompts
A key finding of the study is the superiority of image-based prompts over text-based ones in the medical domain. In zero-shot settings, PBIP significantly outperformed text-prompted models. This suggests that the visual complexity of histopathology—where textures and colors carry more weight than categorical names—is better captured through representative visual examples than through linguistic descriptions.
Discussion and Limitations
While PBIP represents a significant advancement, it is not without limitations. The performance of the image bank is sensitive to the choice of $K$ (the number of clusters). Optimal results typically require $K$ values between 2 and 4, which currently necessitates manual tuning per dataset. Furthermore, the framework's efficacy is tied to the quality of the underlying MedCLIP encoder; its ability to segment rare or novel tissue types is limited by the encoder's pre-training exposure.
Conclusion
The Prototype-Based Image Prompting framework demonstrates that visual prototypes are a powerful tool for weakly supervised histopathological segmentation. By capturing intra-class diversity and enforcing inter-class distinction through contrastive learning, PBIP overcomes the primary hurdles of traditional CAM-based and text-based methods. Future work will focus on automating prototype selection and exploring multi-modal prompting strategies that combine the strengths of both images and expert-curated text.
Summary updated