Weakly-Supervised Salient Object Detection Using Point Supervision

AAAI 2022

TL;DR

This paper introduces a novel approach to salient object detection (SOD) using only single-point annotations, significantly reducing annotation effort. They propose a new framework that leverages adaptive flood filling for pseudo-label generation and a transformer-based model, enhanced by a Non-Salient Suppression (NSS) module to refine predictions and achieve state-of-the-art results, even surpassing some fully supervised methods.

What Problem Does This Paper Address?

Salient object detection (SOD) aims to identify the most visually prominent regions in an image, mimicking human attention. Traditional SOD models achieve high accuracy but rely on large datasets with pixel-wise annotations, which are extremely time-consuming and expensive to create. While weakly supervised methods using image-level labels, bounding boxes, or scribbles have emerged to reduce annotation burden, point-level supervision, which is the least labor-intensive, has remained largely unexplored for SOD. A key challenge with sparse supervision is that models tend to highlight all detected objects, failing to distinguish between salient and non-salient ones, leading to a general foreground detection rather than true saliency.

Methods

The authors propose a two-stage training framework:

  1. Pseudo-Label Generation:
    • Adaptive Masked Flood Filling: They use existing edge detectors to find image edges and then employ a flood-filling algorithm, guided by these edges and an adaptive circular mask, to generate initial "pseudo-labels" from the sparse point annotations. The adaptive mask's radius is dynamically adjusted based on image dimensions to handle edge discontinuities.
  2. Transformer-Based Point-Supervised Model: A transformer architecture (specifically, a ResNet-50 backbone with a ViT-Base encoder) is used to process the image and generate initial saliency maps based on these pseudo-labels.
  3. Non-Salient Suppression (NSS) and Refinement:
    • NSS Module: To address the issue of models detecting non-salient objects, they introduce the NSS module. This module uses the generated saliency maps from the first round and the original point annotations to explicitly suppress non-salient regions, creating refined pseudo-labels for the second training round.
  4. Second Training Round: The refined pseudo-labels are used to further train the transformer model, leading to improved saliency maps.

The model also incorporates an "Edge-preserving Decoder" to help the model learn structural cues from the sparse annotations. The loss function includes binary cross-entropy for edge prediction, partial cross-entropy for saliency, and a gated CRF loss for structural consistency.

Datasets

  • P-DUTS: A new point-supervised dataset created by relabeling the existing DUTS dataset. For each salient object, only a single pixel is annotated.
  • Evaluation Datasets: The model is evaluated on five public benchmark datasets: ECSSD, PASCAL-S, DUT-O, HKU-IS, and DUTS-test.

Key Insights and Ideas

  • Point Supervision is Viable for SOD: The paper demonstrates that even extremely sparse point annotations can be effectively used to train a high-performing SOD model.
  • The "General Foreground Detector" Problem: A crucial insight is that weakly supervised models with sparse labels often fail to distinguish between salient and non-salient objects, leading to a general foreground detection problem.
  • NSS as a Solution: The Non-Salient Suppression (NSS) module is a key innovation that directly tackles this problem by explicitly filtering out non-salient regions, significantly improving the quality of saliency maps.
  • Adaptive Flood Filling for Pseudo-Labels: The adaptive flood-filling approach effectively leverages sparse points and edge information to create meaningful initial pseudo-labels, overcoming limitations of simple point-based methods.

Key Results

  • The proposed method achieves state-of-the-art performance among weakly supervised SOD methods, often by a significant margin.
  • On several benchmarks, the model even surpasses the performance of many fully supervised methods. For instance, on the DUTS-test dataset, their method achieves an Fmax of 0.8586, which is competitive with or better than several fully supervised models listed in Table 1.
  • Qualitative results (Figure 6) show that their method produces more accurate and complete saliency maps, effectively suppressing non-salient objects and capturing fine details.

Limitations and Future Work

  • Annotation Effort vs. Performance Trade-off: While point supervision is the least labor-intensive, the paper acknowledges that the performance might still be slightly lower than methods using stronger supervision (though their results challenge this).
  • Hyperparameter Sensitivity: The paper briefly mentions the impact of the hyperparameter $\gamma$ in the adaptive flood filling, noting that too small or too large values can negatively affect performance.
  • Future Work: The paper doesn't explicitly detail future work, but the success of point supervision opens avenues for exploring even sparser or more complex annotation schemes for SOD and other dense prediction tasks. The transformer architecture also suggests potential for further exploration in this domain.

Summary updated

BibTeX
@inproceedings{2022weakly,
  title = {{Weakly-Supervised Salient Object Detection Using Point Supervision}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
}
Figure