TL;DR
This paper introduces Shortcut Mitigating Augmentation (SMA), a novel data augmentation technique for weakly supervised semantic segmentation (WSSS). SMA disentangles object and background features, then shuffles them to create synthetic training examples that break spurious correlations, leading to more robust and accurate object localization.
What Problem Does This Paper Address?
Semantic segmentation, the task of assigning a class label to every pixel in an image, typically requires expensive pixel-level annotations. Weakly supervised semantic segmentation (WSSS) aims to reduce this cost by using weaker labels, such as image-level class labels. However, WSSS models trained on image-level labels often learn "shortcuts" – spurious correlations between objects and their common backgrounds (e.g., "sheep" always appearing on "grass"). This leads to poor generalization, where the model fails when objects appear in unusual contexts. This paper tackles the problem of these learned shortcuts, which hinder the performance of WSSS models.
Methods
The core of the proposed method is Shortcut Mitigating Augmentation (SMA). It operates in two main stages:
- Disentangling Object-Relevant and Background Features:
- Instead of standard global average pooling, SMA uses specialized aggregators ($M_o$, $M_b$) to extract object-relevant ($z^o$) and background ($z^b$) features from the backbone network's feature maps.
- These features are further separated using a contrastive loss ($L_{contr}$) that encourages the distance between object and background representations to increase.
- A classification loss ($L_{cls}$) is applied, where $z^o$ is supervised with the true class label ($y$) and $z^b$ is supervised with a zero vector, forcing $z^b$ to represent non-object attributes.
- Shuffling Disentangled Representations:
- To create synthetic training data, SMA randomly shuffles the disentangled background representations ($z^b$) within a mini-batch.
- These shuffled background features are then concatenated with the original object features ($z^o$) to form a new representation ($z^{sb}$). This synthetic representation pairs the object with an unseen background.
- The process is also performed in the opposite direction (shuffling $z^o$ and concatenating with $z^b$) to create $z^{osb}$.
- A shuffle loss ($L_{shuffle}$) is used to train the classifier on these novel, shuffled representations. Crucially, the background features are detached from the backpropagation of the shuffle loss to prevent learning from potentially incorrect shuffled background signals.
The overall objective function combines the classification loss, contrastive loss, and shuffle loss.
Datasets
- Pascal VOC 2012: Used for training and evaluation. This dataset contains 20 object categories and 1 background class.
- MS COCO 2014: Used to demonstrate the generalization ability and robustness of the proposed method. This dataset is larger, with 81 classes (including background).
Key Insights and Ideas
- Shortcuts are a major bottleneck: The paper convincingly argues that spurious correlations between objects and their typical backgrounds are a primary reason for poor generalization in WSSS.
- Disentanglement is key: Separating object and background features is crucial for breaking these shortcuts. SMA's specialized aggregators and contrastive loss are designed for this.
- Synthetic data breaks biases: By shuffling and recombining features from different images, SMA creates novel object-background pairings that are unlikely to exist in the original biased dataset, forcing the model to rely on object-specific features.
- Analyzing shortcut usage is important: The paper introduces a metric (Shortcut Usage Ratio - SUR) based on Integrated Gradients to quantify how much a model relies on background vs. object features, providing a way to evaluate the effectiveness of augmentation methods in mitigating shortcuts.
Key Results
- Improved Localization: When applied to existing WSSS methods (PSA, IRN, AdvCAM, AMN), SMA consistently improved the mIoU for both initial seed localization maps and pseudo-masks on Pascal VOC 2012. For example, applying SMA to AMN resulted in an mIoU of 72.7% for pseudo-masks.
- Enhanced Semantic Segmentation: SMA also boosted the final semantic segmentation performance, with improvements seen on both Pascal VOC 2012 and MS COCO 2014. For instance, on MS COCO 2014, SMA improved AMN by 1.2%p.
- Reduced Shortcut Reliance: The SUR analysis showed that SMA achieved the highest SUR across various object-background pairs, indicating that the classifier trained with SMA relies less on background cues and more on object-relevant features.
- Ablation Studies: The paper demonstrated that each component of SMA (classification loss, contrastive loss, and shuffle loss) contributes to performance improvements, with the full SMA method yielding the best results.
Limitations and Future Work
- Two-step process: SMA requires an additional training stage, increasing training time.
- Decoupling hard attributes: The method faces challenges in effectively decoupling very hard-to-separate attributes.
- Future Work: The authors suggest combining representations based on distance to allow the classifier to leverage more object-related information for improved localization. They also aim to address the limitations of effectively decoupling hard attributes.
Summary updated