Abstract
Weakly Supervised Semantic Segmentation (WSSS) aims to bridge the gap between image-level labels and pixel-level masks, reducing the need for expensive manual annotation. Traditional Class Activation Maps (CAMs) often fail by focusing only on the most discriminative object parts. This paper introduces the Embedded Discriminative Attention Mechanism (EDAM), which integrates mask generation into the classification process and utilizes collaborative attention across images to capture full object extents. EDAM achieves state-of-the-art results, reaching 70.6% mIoU on the PASCAL VOC 2012 benchmark.
Introduction
Semantic segmentation usually requires expensive, pixel-perfect manual annotations. WSSS attempts to train models using only image-level tags (e.g., "this image contains a car"). The standard tool for this is the Class Activation Map (CAM). However, CAMs have a fundamental flaw: a classifier only needs to find the most "discriminative" part of an object to label it correctly (like a bird’s beak or a car’s wheel). For segmentation, we need the entire object. Previous attempts to fix this—such as randomly erasing parts of the image—are often complex and introduce significant noise.
The EDAM Framework
The authors propose EDAM, which consists of two primary novel components integrated into a standard backbone like ResNet:
- Discriminative Activation (DA) Layer: Instead of generating heatmaps as a post-processing step, the DA layer is baked into the network. It explicitly produces $K+1$ normalized masks (for $K$ categories plus one for the background). By explicitly modeling the background, the network reduces "scattering" where foreground features bleed into the rest of the image.
- Collaborative Multi-Attention (CMA) Module: This is the core innovation used during training. It takes the activation maps from a batch of images and applies a self-attention mechanism to explore inter-image homogeneity. If one image shows a cat's head and another shows a cat's tail, the CMA module helps the network realize that both features belong to the same semantic category, leading to a more comprehensive mask.
Crucially, the CMA module is only used during training. During inference, the model is streamlined, using only the DA layer to produce pseudo-labels for segmentation, ensuring computational efficiency.
Key Insights
- From Local to Global: The transition from "what part makes this a car?" to "what are all the parts of a car?" is achieved by looking at multiple instances of the category simultaneously.
- Explicit Backgrounding: Most WSSS methods treat the background as a byproduct. EDAM treats the background as its own class within the DA layer, which significantly cleans up the resulting masks.
- The Power of Two: Interestingly, the authors found that collaborative information from just two images in a batch was enough to significantly improve results; adding more images yielded diminishing returns.
Experimental Results
The model was evaluated on the PASCAL VOC 2012 and COCO-Stuff 10k datasets. EDAM achieved 70.9% mIoU on the PASCAL VOC 2012 validation set and 70.6% on the test set, outperforming previous leaders like ICD and MCIS by roughly 2.6–3.1%.
Ablation studies confirmed the effectiveness of the individual components: adding the DA layer improved the baseline CAM mIoU by over 4%, while the CMA module added another 1.2% on top of that. Visualizations show that EDAM heatmaps cover significantly more of the object area (e.g., the entire body of a cow or horse) compared to the sparse "hotspots" produced by traditional CAMs.
Conclusion
EDAM successfully addresses the limitations of CAM-based weakly supervised segmentation by embedding discriminative attention directly into the network architecture. By leveraging collaborative information across images, it produces more complete pseudo-labels. While the method still relies on external saliency maps for background refinement, it sets a new performance standard for the field. Future work may explore integrating cross-image relationships without such external dependencies.
Summary updated