Cobra: Complementary Branch Fusing Class and Semantic Knowledge for Robust Weakly Supervised Semantic Segmentation

Woojung Han, Seil Kang, Kyobin Choo, Seong Jae Hwang

TL;DR

This paper introduces CoBra, a novel dual-branch framework that synergizes the strengths of Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) for weakly supervised semantic segmentation (WSSS). By fusing class-specific localization from CNNs with semantic understanding from ViTs, CoBra generates more accurate and robust pseudo-masks, achieving state-of-the-art results on challenging benchmarks.

What Problem Does This Paper Address?

Semantic segmentation, the task of assigning a class label to every pixel in an image, typically requires extensive pixel-level annotations. Acquiring these labels is labor-intensive and costly. Weakly supervised semantic segmentation (WSSS) aims to alleviate this by using weaker forms of supervision, such as image-level labels (e.g., "this image contains a cat").

While Class Activation Maps (CAMs) from CNNs provide strong class-specific localization (e.g., focusing on a cat's face), they often miss broader object parts. Conversely, Vision Transformers (ViTs), with their self-attention mechanisms, excel at capturing semantic relationships and object outlines but struggle with precise class discrimination, often localizing multiple objects or background elements. This paper addresses the challenge of combining these complementary strengths to produce more accurate and complete object masks from image-level supervision.

Methods

The core of CoBra is a dual-branch framework that leverages both CNNs and ViTs.

  • Class-Aware Knowledge (CAK) Branch: This branch uses a CNN backbone to generate CNN CAMs. These CAMs are good at identifying class-specific regions but lack semantic completeness. The branch also produces pseudo-labels by applying an argmax to the CNN CAMs.
  • Semantic-Aware Knowledge (SAK) Branch: This branch employs a ViT backbone. ViTs generate ViT CAMs that capture semantic relationships and object boundaries well but can be imprecise in class localization. A key output here is patch affinity, derived from ViT's self-attention, which describes semantic relationships between image patches.

The novelty lies in the complementary branch losses that enable knowledge exchange between the two branches:

  • Class-Aware Projection (CAP): The CAK branch generates CAP, which is then refined by the semantic-aware knowledge from the SAK branch's patch affinity. This loss (Lcap) pulls semantically similar patches (identified by SAK) towards the target patch in the class feature space, improving the CNN's semantic sensitivity.
  • Semantic-Aware Projection (SAP): The SAK branch generates SAP, which is refined by the class-aware knowledge from the CAK branch's pseudo-labels. This loss (Lsap) pushes away incorrectly localized patches (identified by CAK's class precision) in the semantic feature space, improving the ViT's class precision.

These complementary losses, along with standard classification and CAM consistency losses (Lcls and Lcam), are combined into a total loss function (Ltotal) to train the dual-branch model. Finally, the generated seeds (CAMs) from both branches are fused using a max operation to create the final pseudo-mask for segmentation.

Datasets

The authors evaluate their method on two widely used datasets:

  • PASCAL VOC 2012: Contains 20 foreground classes and 1 background class.
  • MS COCO 2014: Features 80 object classes and 1 background class.

They also mention using an augmented training set for PASCAL VOC 2012, comprising 10,582 images.

Key Insights and Ideas

  • CNNs and ViTs are complementary: The paper strongly demonstrates that CNNs excel at class precision (knowing what is there) while ViTs excel at semantic sensitivity (knowing where it is and its context). Neither is sufficient alone for robust WSSS.
  • Targeted knowledge infusion: The proposed CAP and SAP losses are crucial. They don't just average features; they actively guide one branch with the strengths of the other, specifically addressing their respective weaknesses (CNN's lack of semantic coverage, ViT's lack of class specificity).
  • Patch affinity as a powerful semantic cue: Leveraging ViT's patch affinity matrix proves effective in guiding the CNN branch to consider broader semantic context beyond just the most discriminative parts.
  • Pseudo-labels are key, but their quality matters: The paper emphasizes that the quality of the generated pseudo-labels is paramount. By fusing the strengths of both CNN and ViT, CoBra generates superior pseudo-labels that lead to better final segmentation.
  • Minimal hyperparameter sensitivity: The authors note that their model's performance remains consistently high even with variations in the lambda values for the complementary losses, suggesting robustness.

Key Results

  • State-of-the-art performance: CoBra achieves state-of-the-art results on PASCAL VOC 2012, reaching 74.0% mIoU on the validation set and 73.9% on the test set with a ResNet101 backbone. For the MiT-B2 backbone, it achieves 74.3% and 74.2% respectively.
  • Improved seed and mask quality: Without CRF post-processing, CoBra's mask achieves 71.9% mIoU, outperforming existing methods that include CRF. With CRF, it reaches 73.5% mIoU, demonstrating superior seed and mask generation.
  • MS COCO 2014 performance: The method also shows impressive results on MS COCO 2014, achieving 45.5% mIoU with a ResNet101 backbone, which is competitive.
  • Qualitative improvements: Visual results show CoBra's ability to generate more complete object masks with accurate boundaries, effectively differentiating between classes and reducing false positives/negatives compared to individual branches.

Limitations and Future Work

  • Computational cost: While not explicitly stated as a limitation, dual-branch architectures inherently increase computational complexity compared to single-branch models.
  • Hyperparameter tuning for k: The paper mentions an ablation study for ksap and keap (parameters controlling the number of positive/negative patches used in losses), suggesting that optimal values might require careful tuning. However, they found that leveraging a small number of highly confident patches (e.g., top 10%) yields robustness.
  • Future work: The authors suggest that their work provides insights into the complementary nature of CNNs and ViTs, encouraging further exploration of how these architectures can be jointly leveraged for various computer vision tasks. The paper also highlights the potential for further improvements by exploring different fusion strategies and loss functions.

Summary updated

Abstract

Leveraging semantically precise pseudo masks derived from image-level class knowledge for segmentation, namely image-level Weakly Supervised Semantic Segmentation (WSSS), still remains challenging. While Class Activation Maps (CAMs) using CNNs have steadily been contributing to the success of WSSS, the resulting activation maps often narrowly focus on class-specific parts (e.g., only face of human). On the other hand, recent works based on vision transformers (ViT) have shown promising results based on their self-attention mechanism to capture the semantic parts but fail in capturing complete class-specific details (e.g., entire body parts of human but also with a dog nearby). In this work, we propose Complementary Branch (CoBra), a novel dual branch framework consisting of two distinct architectures which provide valuable complementary knowledge of class (from CNN) and semantic (from ViT) to each branch. In particular, we learn Class-Aware Projection (CAP) for the CNN branch and Semantic-Aware Projection (SAP) for the ViT branch to explicitly fuse their complementary knowledge and facilitate a new type of extra patch-level supervision. Our model, through CoBra, fuses CNN and ViT's complementary outputs to create robust pseudo masks that integrate both class and semantic information effectively. Extensive experiments qualitatively and quantitatively investigate how CNN and ViT complement each other on the PASCAL VOC 2012 dataset, showing a state-of-the-art WSSS result. This includes not only the masks generated by our model, but also the segmentation results derived from utilizing these masks as pseudo labels.
BibTeX
@misc{han2024cobra,
  author = {Woojung Han and Seil Kang and Kyobin Choo and Seong Jae Hwang},
  title = {{Cobra: Complementary Branch Fusing Class and Semantic Knowledge for Robust Weakly Supervised Semantic Segmentation}},
  year = {2024},
  eprint = {2403.08801},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.08801},
}
Figure