CPEWS: Contextual Prototype-Based End-to-End Weakly Supervised Semantic Segmentation

Xiaoyan Shao, Jiaqi Han, Lingling Li, Xuezhuan Zhao, Jingjing Yan2025 · DOI

Abstract

Weakly Supervised Semantic Segmentation (WSSS) aims to bridge the performance gap with fully supervised methods by using cheaper annotations like image-level tags. The CPEWS framework introduces an end-to-end approach leveraging Vision Transformers (ViT) and a novel contextual prototype system. By addressing the fundamental flaws of Class Activation Maps (CAMs)—incomplete object coverage and background over-activation—CPEWS achieves state-of-the-art results on benchmark datasets without requiring additional supervision like saliency maps.

The Challenge of Weakly Supervised Segmentation

Semantic segmentation typically requires pixel-level "ground truth" labels, which are labor-intensive to produce. WSSS offers a cost-effective alternative by training models on image-level tags (e.g., "cat," "car"). Traditionally, these methods rely on Class Activation Maps (CAMs) to localize objects. However, CAMs suffer from two critical limitations:

  1. Incomplete Activation: The model often focuses only on the most discriminative parts of an object, such as a bird's head while ignoring its body.
  2. False Activation: The model may incorrectly associate objects with frequently co-occurring background elements, such as a runway being activated alongside an airplane.
Common issues in Class Activation Maps (CAMs): (a) Incomplete activation and (b) False activation.

While Convolutional Neural Networks (CNNs) have been the standard for these tasks, their inherent local receptive fields often fail to capture the global context necessary to resolve these ambiguities.

Methodology: The CPEWS Framework

CPEWS (Contextual Prototype-Based End-to-End) departs from the traditional multi-stage WSSS pipeline, opting for a more efficient single-stage training process.

Vision Transformer (ViT) Backbone

Unlike CNNs, the ViT architecture utilizes self-attention mechanisms to capture global dependencies. This allows the model to perceive the entire object simultaneously, mitigating the issue of incomplete activation.

Intermediate Supervised Module (ISM)

The researchers observed that intermediate layers in a ViT often contain cleaner semantic information than the final layer, which can become "over-smoothed." The ISM uses features from these intermediate layers to supervise the final output via cosine similarity, effectively sharpening object boundaries.

Contextual Prototype Module (CPM)

The CPM generates "prototypes"—representative feature vectors—for both foreground objects and background/co-occurring elements. This transforms segmentation into a similarity-matching problem: pixels are classified based on their proximity to these prototypes in the feature space.

Specialized Loss Functions

  • Prototype Discrimination Loss ($L_{PDL}$): This loss function pulls object features toward their respective positive prototypes while pushing them away from negative (background) prototypes.
  • Superclass Suppression Loss ($L_{SSL}$): Designed to combat false activations, this loss uses superclass information to prevent the model from suppressing parts of an object simply because they resemble a related category (e.g., distinguishing a bus from a train).

Experimental Results

CPEWS was evaluated on the PASCAL VOC 2012 and MS COCO 2014 benchmarks, setting new performance standards for end-to-end WSSS methods.

  • PASCAL VOC 2012: The model achieved a Mean Intersection over Union (MIoU) of 69.8% on the validation set and 72.6% on the test set, outperforming the previous state-of-the-art method, ToCo, by 2.1% on the test set.
  • MS COCO 2014: On this larger and more complex dataset, CPEWS reached 41.4% MIoU, demonstrating its scalability to multi-object scenes.

Conclusion and Future Directions

CPEWS demonstrates that end-to-end training with global context and prototype-based learning can significantly narrow the gap between weakly and fully supervised segmentation. While the model is computationally intensive due to prototype updates and its large parameter count (approx. 99 million), it provides a robust foundation for future research. Potential improvements include model pruning for edge deployment and the integration of advanced data augmentation techniques to further refine performance.

Summary updated

Abstract

The primary challenge in weakly supervised semantic segmentation is effectively leveraging weak annotations while minimizing the performance gap compared to fully supervised methods. End-to-end model designs have gained signi... | Find, read and cite all the research you need on Tech Science Press
BibTeX
@inproceedings{shao2025cpews,
  author = {Xiaoyan Shao and Jiaqi Han and Lingling Li and Xuezhuan Zhao and Jingjing Yan},
  title = {{CPEWS: Contextual Prototype-Based End-to-End Weakly Supervised Semantic Segmentation}},
  booktitle = {Computers,},
  year = {2025},
  doi = {10.32604/cmc.2025.060295},
}
Figure