MoRe: Class Patch Attention Needs Regularization for Weakly Supervised Semantic Segmentation

Zhiwei Yang, Yucong Meng, Kexue Fu, Shuo Wang, Zhijian Song

Introduction

Semantic segmentation is a cornerstone of computer vision, yet its reliance on pixel-level annotations makes it prohibitively expensive for large-scale applications. Weakly Supervised Semantic Segmentation (WSSS) addresses this by training models using only image-level labels. While early methods relied on Class Activation Maps (CAMs), recent shifts toward Vision Transformers (ViTs) have introduced Localization Attention Maps (LAMs) to capture broader global context.

However, LAMs suffer from a significant artifact issue. Without strict constraints, class tokens often "falsely activate" background patches that lack semantic relevance, leading to noisy and inaccurate segmentation masks.

The artifact issue in standard ViT-based WSSS, where class tokens falsely activate background patches without regularization.

The MoRe Framework

To combat these artifacts, the authors propose MoRe (Regularized Class-Patch Attention), a framework that imposes both implicit and explicit regularization on the attention mechanism within ViTs.

1. Graph Category Representation (GCR)

Rather than treating attention as a flat matrix, GCR conceptualizes it as a directed graph. By representing class and patch tokens as nodes and utilizing "head" and "tail" embeddings, the model can more effectively filter interactions. GCR dynamically selects only the top-K most similar neighbors, implicitly pruning low-information "artifact" patches during the feature aggregation process.

2. Localization-informed Regularization (LIR)

LIR leverages the inherent spatial smoothness of CAMs to explicitly supervise the noisier LAMs. This module employs two distinct loss functions:

  • Confident Relation Enhancement ($L_{cre}$): A contrastive loss that aligns class tokens with "confident" foreground patches identified by CAM while repelling unrelated background regions.
  • Uncertain Relation Enhancement ($L_{ure}$): This module uses a mining kernel to identify uncertain but potentially relevant regions, encouraging the model to expand its attention to the full spatial extent of the object.
The MoRe framework applies regularization to class-patch attention, producing significantly cleaner and more accurate localization maps.

Experimental Results

MoRe was evaluated on the PASCAL VOC 2012 and MS COCO 2014 benchmarks, demonstrating superior performance and efficiency.

  • State-of-the-Art Accuracy: MoRe achieved 76.4% mIoU on PASCAL VOC, outperforming existing single-stage methods. On the more challenging MS COCO dataset, it reached 47.4% mIoU, a substantial improvement over the previous state-of-the-art (DuPL at 44.6%).
  • Training Efficiency: MoRe is highly optimized for speed. It completes the WSSS workflow in 372 minutes on an RTX 3090, nearly four times faster than the previous leading method, MCTformer+ (1496 minutes), while requiring less GPU memory (12.1 GB vs. 18.0 GB).
  • Qualitative Superiority: Visualizations confirm that MoRe captures fine structural details—such as boat masts and chair legs—that are typically lost or blurred by non-regularized methods.

Conclusion

MoRe demonstrates that the "artifact issue" in ViT-based WSSS is not a result of over-smoothing but of improper correlation. By treating attention as a topological graph and using CAMs as a supervisory signal, MoRe bridges the gap between global context and local precision, setting a new standard for efficient, single-stage weakly supervised segmentation.

Summary updated

Abstract

Weakly Supervised Semantic Segmentation (WSSS) with image-level labels typically uses Class Activation Maps (CAM) to achieve dense predictions. Recently, Vision Transformer (ViT) has provided an alternative to generate localization maps from class-patch attention. However, due to insufficient constraints on modeling such attention, we observe that the Localization Attention Maps (LAM) often struggle with the artifact issue, i.e., patch regions with minimal semantic relevance are falsely activated by class tokens. In this work, we propose MoRe to address this issue and further explore the potential of LAM. Our findings suggest that imposing additional regularization on class-patch attention is necessary. To this end, we first view the attention as a novel directed graph and propose the Graph Category Representation module to implicitly regularize the interaction among class-patch entities. It ensures that class tokens dynamically condense the related patch information and suppress unrelated artifacts at a graph level. Second, motivated by the observation that CAM from classification weights maintains smooth localization of objects, we devise the Localization-informed Regularization module to explicitly regularize the class-patch attention. It directly mines the token relations from CAM and further supervises the consistency between class and patch tokens in a learnable manner. Extensive experiments on PASCAL VOC and MS COCO validate that MoRe effectively addresses the artifact issue and achieves state-of-the-art performance, surpassing recent single-stage and even multi-stage methods.
BibTeX
@inproceedings{yang2025more,
  author = {Zhiwei Yang and Yucong Meng and Kexue Fu and Shuo Wang and Zhijian Song},
  title = {{MoRe: Class Patch Attention Needs Regularization for Weakly Supervised Semantic Segmentation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i9.33018},
}
Figure