TL;DR
This paper introduces Representation Mutual Learning (RML), a novel decoder-free framework for weakly supervised semantic segmentation (WSSS) that directly predicts segmentation masks. By leveraging mutual learning across instance, feature, and pixel levels, RML significantly boosts segmentation accuracy without the overhead of traditional decoders.
What Problem Does This Paper Address?
Semantic segmentation, the task of assigning a class label to every pixel in an image, typically requires extensive pixel-level annotations. Acquiring these labels is labor-intensive and costly. Weakly Supervised Semantic Segmentation (WSSS) aims to alleviate this by using cheaper, less precise labels like image-level tags. While recent end-to-end WSSS methods have shown promise, they often rely on segmentation branches or decoders, increasing model complexity and inference time. This paper tackles the challenge of achieving high WSSS performance with an efficient, end-to-end, decoder-free approach.
Methods
The core of this work is the Representation Mutual Learning (RML) framework. Unlike traditional methods that use decoders to refine initial predictions, RML directly learns segmentation masks by enabling different levels of feature representations to learn from each other. This is achieved through three key strategies:
- CAM-driven Instance-leave Mutual Learning (CIML): This strategy focuses on improving the quality of Class Activation Maps (CAMs), which are initial heatmaps indicating object presence. CIML ensures that CAMs are "equivariant" (transform consistently with image transformations) and increases the separation between semantic prototypes of different object classes.
- Multi-scale Feature-leave Mutual Learning (MFML): This component aims to enhance the understanding of contextual information within an image. It uses multi-scale features from different network layers, aligns their aggregated representations, and uses mutual information to preserve specific details while learning contextual relationships.
- Affinity-aware Pixel-level Mutual Learning (APML): This strategy focuses on learning semantic affinity, which describes the relationship between adjacent pixels. It combines pixel-level mutual information loss with an affinity-aware contrastive learning loss to refine semantic affinity representations.
These three strategies work together, guided by a classification loss, to directly predict segmentation masks.
Datasets
The proposed RML framework is evaluated on two standard benchmarks for WSSS:
- PASCAL VOC 2012: A dataset with 20 foreground object classes and 1 background class. The authors use an augmented version with the SBD dataset for training.
- MS COCO 2014: A larger dataset comprising 80 foreground object classes and 1 background class.
The paper uses image-level labels as supervision for both datasets.
Key Insights and Ideas
- Decoder-free is Viable: The paper demonstrates that it's possible to achieve state-of-the-art WSSS performance without relying on explicit segmentation branches or decoders, simplifying the model architecture and reducing computational cost.
- Mutual Learning Across Levels is Powerful: The success of RML hinges on the synergistic interaction between instance-level (CAMs), feature-level (context), and pixel-level (affinity) representations. Each level contributes uniquely to improving the final segmentation.
- CAM Quality is Crucial: Enhancing the initial CAMs through instance-level mutual learning directly translates to better segmentation masks, highlighting the importance of these initial cues.
- Context and Affinity are Key Complementors: MFML and APML address the limitations of relying solely on local features by incorporating broader contextual understanding and fine-grained pixel relationships.
Key Results
On the PASCAL VOC 2012 validation set, RML achieves a mean Intersection over Union (mIoU) of 67.2%, significantly outperforming previous end-to-end methods. When combined with CRF post-processing, this further improves to 67.5%. On the MS COCO 2014 validation set, RML achieves 40.0% mIoU, also demonstrating strong performance against existing methods. The paper also highlights that RML is computationally more efficient than some prior methods, taking about 3 hours on a single NVIDIA 3090 GPU for the VOC dataset.
Limitations and Future Work
- Post-processing: While RML achieves strong results, the performance is further boosted by CRF post-processing. Future work could aim to integrate such refinement capabilities directly into the end-to-end framework.
- Hyperparameter Sensitivity: The paper mentions providing a sensitivity analysis for the loss weighting hyperparameters ($\alpha, \beta_1, \beta_2$) in the appendix, suggesting that tuning these might be important for optimal performance.
- Generalization: While evaluated on standard benchmarks, exploring RML's performance on more diverse datasets or in different WSSS settings (e.g., scribbles, bounding boxes) could be a future direction.
The authors suggest that their RML framework provides a strong foundation for future research in efficient and effective WSSS.
Summary updated