Introduction
Semantic segmentation is a cornerstone of computer vision, yet its reliance on pixel-level annotations makes it prohibitively expensive for large-scale applications. Weakly Supervised Semantic Segmentation (WSSS) addresses this by training models using only image-level labels. While early methods relied on Class Activation Maps (CAMs), recent shifts toward Vision Transformers (ViTs) have introduced Localization Attention Maps (LAMs) to capture broader global context.
However, LAMs suffer from a significant artifact issue. Without strict constraints, class tokens often "falsely activate" background patches that lack semantic relevance, leading to noisy and inaccurate segmentation masks.
The MoRe Framework
To combat these artifacts, the authors propose MoRe (Regularized Class-Patch Attention), a framework that imposes both implicit and explicit regularization on the attention mechanism within ViTs.
1. Graph Category Representation (GCR)
Rather than treating attention as a flat matrix, GCR conceptualizes it as a directed graph. By representing class and patch tokens as nodes and utilizing "head" and "tail" embeddings, the model can more effectively filter interactions. GCR dynamically selects only the top-K most similar neighbors, implicitly pruning low-information "artifact" patches during the feature aggregation process.
2. Localization-informed Regularization (LIR)
LIR leverages the inherent spatial smoothness of CAMs to explicitly supervise the noisier LAMs. This module employs two distinct loss functions:
- Confident Relation Enhancement ($L_{cre}$): A contrastive loss that aligns class tokens with "confident" foreground patches identified by CAM while repelling unrelated background regions.
- Uncertain Relation Enhancement ($L_{ure}$): This module uses a mining kernel to identify uncertain but potentially relevant regions, encouraging the model to expand its attention to the full spatial extent of the object.
Experimental Results
MoRe was evaluated on the PASCAL VOC 2012 and MS COCO 2014 benchmarks, demonstrating superior performance and efficiency.
- State-of-the-Art Accuracy: MoRe achieved 76.4% mIoU on PASCAL VOC, outperforming existing single-stage methods. On the more challenging MS COCO dataset, it reached 47.4% mIoU, a substantial improvement over the previous state-of-the-art (DuPL at 44.6%).
- Training Efficiency: MoRe is highly optimized for speed. It completes the WSSS workflow in 372 minutes on an RTX 3090, nearly four times faster than the previous leading method, MCTformer+ (1496 minutes), while requiring less GPU memory (12.1 GB vs. 18.0 GB).
- Qualitative Superiority: Visualizations confirm that MoRe captures fine structural details—such as boat masts and chair legs—that are typically lost or blurred by non-regularized methods.
Conclusion
MoRe demonstrates that the "artifact issue" in ViT-based WSSS is not a result of over-smoothing but of improper correlation. By treating attention as a topological graph and using CAMs as a supervisory signal, MoRe bridges the gap between global context and local precision, setting a new standard for efficient, single-stage weakly supervised segmentation.
Summary updated