3SD: Self-Supervised Saliency Detection With No Labels

Rajeev Yasarla, Renliang Weng, Wongun Choi, Vishal M. Patel, Amir SadeghianWACV 2024

TL;DR

This paper introduces 3SD, a novel self-supervised method for salient object detection that eliminates the need for any human-annotated labels. By cleverly generating "pseudo-ground truth" labels from class activation maps (CAMs) derived from a patch-wise contrastive learning task, 3SD achieves performance on par with, and sometimes exceeding, fully-supervised methods, while outperforming existing weakly and unsupervised approaches.

What Problem Does This Paper Address?

Salient object detection (SOD) aims to identify and segment the most visually attention-grabbing regions in an image. Traditional methods often rely on extensive pixel-wise annotations, which are time-consuming and expensive to create. While weakly supervised methods (using image-level labels or scribbles) and unsupervised methods (relying on handcrafted features or noisy pseudo-labels) have emerged to reduce annotation burden, they often struggle to produce high-quality saliency maps, particularly with fine details and sharp boundaries. This paper addresses the critical need for a highly effective SOD method that completely bypasses the requirement for any form of human annotation, including weak labels.

Methods

The core of 3SD lies in its innovative approach to generating pseudo-ground truth (pseudo-GT) labels. The method employs a base network (BN) with a novel encoder-decoder architecture. The encoder comprises a local encoder (EL) for pixel-level relationships and a global encoder (EG) using a transformer for long-range context. The decoder has two branches: one for saliency prediction and another for classification.

The key innovation is in the self-supervised classification task used to train the classification decoder. Instead of a standard image-wise classification, 3SD utilizes a patch-wise contrastive learning paradigm. This involves:

  1. Student-Teacher Framework: A student network is trained to mimic a teacher network (updated via EMA).
  2. Contrastive Loss: Positive patches (similar to salient objects) are pulled closer to negative patches (dissimilar) in the feature space. This forces the network to learn fine-grained semantic information at a patch level.
  3. CAM Generation: The classification decoder outputs class logits maps, from which Class Activation Maps (CAMs) are generated. These CAMs capture semantic information about salient objects.
  4. Pseudo-GT Creation: The CAM maps are then fused with a gated edge map (derived from the input image's edges) to create the pseudo-GT saliency maps. This fusion helps to refine boundaries and improve localization.
  5. Structure-Aware Loss: An additional gated structure-aware loss is incorporated to further enhance boundary estimation.

The entire training process is self-supervised, meaning it does not require any ground truth saliency maps.

Datasets

  • Training Dataset: DUTS-TR (10,553 images) - used for training the base network without its annotations.
  • Evaluation Datasets:
    • DUTS-TE
    • DUT-OMRON
    • HKU-IS
    • PASCAL-S
    • ECSSD

Key Insights and Ideas

  • Patch-wise Contrastive Learning for Semantics: The most significant insight is that learning semantic information at a patch level through contrastive learning is crucial for generating high-quality CAMs, which in turn leads to superior pseudo-GTs. This is a departure from traditional image-wise classification that might focus on discriminative parts rather than the whole object.
  • CAMs as a Bridge to Pseudo-GT: Leveraging CAMs from a classification task as a source of semantic information for saliency detection is a powerful idea. The paper demonstrates that these CAMs, when refined, can effectively guide the generation of pseudo-ground truth.
  • Edge-CAM Fusion for Boundaries: Combining CAMs with edge information is an effective strategy to produce pseudo-GTs with both semantic understanding and sharp boundaries, a common weakness in other unsupervised/weakly supervised methods.
  • Self-Supervision for State-of-the-Art Performance: The paper convincingly shows that a purely self-supervised approach can achieve performance comparable to, or even better than, fully-supervised methods, highlighting the potential of label-free learning in computer vision.

Key Results

  • Outperforms Weakly/Unsupervised Methods: 3SD consistently outperforms all evaluated SOTA weakly and unsupervised methods across five benchmark datasets on metrics like Sm, Fβ, Eη, and MAE.
  • On Par with Fully-Supervised Methods: The performance of 3SD is on par with, and in some cases better than, several SOTA fully-supervised methods. For example, on DUTS-TE, 3SD achieves an Sm score of 0.883, comparable to LDF (0.881) and VST (0.885).
  • Patch Size Matters: The patch-wise contrastive learning approach shows improved performance as patch size increases up to 32x32 pixels, after which performance plateaus.
  • Qualitative Improvements: Visual results demonstrate that 3SD produces sharper, more accurate saliency maps with better boundary adherence compared to other methods, particularly in challenging cases with fine details like horse legs.

Limitations and Future Work

  • Limitations:
    • The paper notes that while the proposed method significantly improves CAM quality, they might not have perfect boundary correspondence, necessitating the fusion with edge maps.
  • The performance of the patch-wise contrastive learning is sensitive to patch size, with 32x32 pixels yielding the best results. Larger patches did not further improve performance.
  • Future Work: The paper suggests that the novel encoder architecture and the patch-wise contrastive learning paradigm for pseudo-GT generation are key contributions that can be extended to other vision tasks. The authors also mention that the student network (fo) in 3SD is used for inference, implying its direct applicability.

Summary updated

BibTeX
@inproceedings{yasarla2024sd,
  author = {Rajeev Yasarla and Renliang Weng and Wongun Choi and Vishal M. Patel and Amir Sadeghian},
  title = {{3SD: Self-Supervised Saliency Detection With No Labels}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure