A Unified Two-Stage Group Semantics Propagation and Contrastive Learning Network for Co-Saliency Detection

Zhenshan Tan, Cheng Chen, Keyu Wen, Yuzhuo Qin, Xiaodong Gu

TL;DR

This paper introduces TopicNet, a novel unified network for co-saliency detection that tackles the challenges of extracting group semantics and suppressing noise objects. By employing a two-stage group semantics propagation mechanism and contrastive learning, TopicNet achieves state-of-the-art performance on multiple benchmarks, demonstrating improved accuracy and robustness.

What Problem Does This Paper Address?

Co-saliency detection (CoSOD) aims to identify multiple related objects that appear repeatedly across a set of images. This is a more complex task than standard salient object detection because it requires not only identifying salient objects but also understanding their shared characteristics and distinguishing them from non-salient or irrelevant objects within the same group. The paper highlights two main challenges:

  1. Group Semantics Extraction: Effectively capturing the common features shared by co-salient objects across different images. Previous methods often struggle with this, leading to scattered and vulnerable group semantics that are easily disrupted by noise. They tend to focus on simple feature aggregation or classification consensus without adequately learning the relevancy between these features.
  2. Noise Object Suppression: Differentiating between truly co-salient objects and those that are salient but not part of the repeating group, or simply background noise. Existing approaches sometimes oversimplify this by completely negating inter-category correlations, which can be detrimental when objects from different classes share visual similarities.

Methods

TopicNet is a unified, end-to-end network composed of two main substructures:

  1. Two-Stage Group Semantics Propagation (TGSP): This module addresses the group semantics extraction challenge.
  2. Image-to-Group Propagation (IGP): This first stage captures consensus representations by propagating similar features within a group of images. It leverages a non-local neural network-inspired approach to learn inter-image correspondences, focusing on identifying shared features across different images in a group.
  3. Group-to-Pixel Propagation (GPP): Following IGP, this second stage builds relevancy by distilling the group semantics to individual pixels. It uses a self-attention mechanism to recalculate pixel-level correlations, enhancing the semantic connection and recalibrating the group semantics by considering subtle features that might have been lost in the first stage.
  4. Contrastive Learning Module (CLM): This module tackles noise object suppression and enhances semantic consistency.
  5. Positive Samples: Designed to pull group semantics closer to adjacent features, reinforcing semantic consistency.
  6. Negative Samples: Carefully designed to push away features of noise objects, thereby suppressing them. The paper proposes three routes for generating negative samples: inter-group objects, element-wise multiplication of non-corresponding features, and a fusion of different attention mechanisms. This approach aims to maximize inter-group object separability without completely ignoring inter-class correlations.

The network is trained end-to-end by jointly optimizing a contrastive loss ($L_{cl}$) and a saliency loss ($L_s$).

Datasets

The paper evaluates TopicNet on three widely used co-saliency detection datasets:

  • CoCA: Contains 80 categories with 1295 images, notable for including at least one noise object in each group.
  • CoSal2015: A challenging dataset with 50 groups and 2015 images.
  • CoSOD3k: A large dataset comprising 160 groups and 3316 images.

Key Insights and Ideas

  • Two-Stage Propagation is Crucial: The paper argues that a single-stage approach to group semantics is insufficient. The two-stage IGP and GPP effectively capture both global group consensus and fine-grained pixel-level relevancy, leading to more robust representations.
  • Contrastive Learning for CoSOD: This is presented as a novel application of contrastive learning to CoSOD. By strategically defining positive and negative samples, it effectively enhances semantic consistency and suppresses noise objects, a key limitation in prior work.
  • Balancing Inter-Class Correlation: Instead of completely discarding inter-class relationships (which can be problematic), TopicNet's contrastive learning aims to maximize separability while still allowing for learning of similar categories, offering a more nuanced approach to noise suppression.
  • Focusing Contrastive Learning on Later Layers: The authors found that applying contrastive learning to the later layers of the encoder (last three layers) is optimal. This preserves crucial low-level features while effectively refining high-level semantic representations, avoiding performance degradation.

Key Results

  • State-of-the-Art Performance: TopicNet outperforms previous state-of-the-art methods across multiple evaluation metrics (maximum F-measure, mean absolute error, average F-measure, maximum E-measure, and S-measure) on the CoCA, CoSal2015, and CoSOD3k datasets.
  • TopicNet-3 vs. TopicNet-2: Using more groups (TopicNet-3) generally yields better performance than using fewer groups (TopicNet-2), though it requires more video memory during training.
  • Real-time Performance: The proposed method achieves real-time inference speeds of around 50 fps on a single GPU, making it practical for downstream applications.
  • Ablation Studies: Experiments confirm the effectiveness of each proposed module (IGP, GPP, CLM) and demonstrate that contrastive learning significantly boosts performance. They also validate the choice of using the last three encoder layers for contrastive learning.

Limitations and Future Work

  • Training Memory: While inference is real-time, using more groups for training (e.g., TopicNet-3) requires substantial video memory.
  • Future Work: The paper doesn't explicitly detail future work directions, but the success of the unified approach and the effectiveness of contrastive learning suggest potential for further exploration in refining group semantics and noise suppression techniques. The real-time performance also opens doors for integration into various video analysis tasks.

Summary updated

Abstract

Co-saliency detection (CoSOD) aims at discovering the repetitive salient objects from multiple images. Two primary challenges are group semantics extraction and noise object suppression. In this paper, we present a unified Two-stage grOup semantics PropagatIon and Contrastive learning NETwork (TopicNet) for CoSOD. TopicNet can be decomposed into two substructures, including a two-stage group semantics propagation module (TGSP) to address the first challenge and a contrastive learning module (CLM) to address the second challenge. Concretely, for TGSP, we design an image-to-group propagation module (IGP) to capture the consensus representation of intra-group similar features and a group-to-pixel propagation module (GPP) to build the relevancy of consensus representation. For CLM, with the design of positive samples, the semantic consistency is enhanced. With the design of negative samples, the noise objects are suppressed. Experimental results on three prevailing benchmarks reveal that TopicNet outperforms other competitors in terms of various evaluation metrics.
BibTeX
@inproceedings{tan2022unified,
  author = {Zhenshan Tan and Cheng Chen and Keyu Wen and Yuzhuo Qin and Xiaodong Gu},
  title = {{A Unified Two-Stage Group Semantics Propagation and Contrastive Learning Network for Co-Saliency Detection}},
  booktitle = {ICME},
  year = {2022},
  doi = {10.1109/icme52920.2022.9859892},
}

Loading related content...

Figure