Few-Cost Salient Object Detection with Adversarial-Paced Learning

Dingwen Zhang, Haibin Tian, Jungong Han

Introduction

Salient Object Detection (SOD)—the task of identifying and segmenting the most visually noticeable objects in an image—is a cornerstone of modern computer vision. However, training state-of-the-art deep learning models for SOD typically requires massive amounts of pixel-wise, human-annotated data. Manually masking thousands of images is an incredibly time-consuming and expensive process.

While semi-supervised and weakly supervised methods exist to alleviate this burden, they often struggle with "noisy" labels. When a model attempts to learn from its own imperfect predictions (pseudo-labels), errors can compound, leading to degraded performance. Existing "Self-Paced Learning" (SPL) techniques attempt to solve this by starting training with "easy" samples and gradually moving to "hard" ones. Unfortunately, these methods rely heavily on heuristic, hand-designed rules to define what constitutes an "easy" sample, which often leads to sub-optimal training curriculums.

To address this, researchers have introduced Few-Cost Salient Object Detection (FC-SOD), a training paradigm that achieves state-of-the-art performance using only a fraction of the manual annotations required by traditional models. At the core of this approach is Adversarial-Paced Learning (APL), which replaces human-designed curriculums with a data-driven adversarial network that learns to identify and trust the most reliable pseudo-labels during training.

The Adversarial-Paced Learning (APL) Framework

The APL framework bridges the gap between Self-Paced Learning (SPL) and Generative Adversarial Learning (GAL) by introducing a learned, rather than calculated, training pace.

1. The Pace-Generator

Instead of using a rigid mathematical formula to decide which samples to learn from, APL introduces a pace-generator. Functioning similarly to a discriminator in a GAN, this network learns to distinguish between real human annotations and the model's own predicted saliency masks. If the pace-generator determines that a predicted mask looks "real," it assigns it a high reliability weight, allowing the main task model to confidently learn from it.

2. Global Structure-Guided Pixel Weighting

A key innovation of the APL framework is the specific architecture of the pace-generator. Saliency is not just about pixel intensity; it relies heavily on object boundaries and coherent shapes. To capture this, the pace-generator consists of two distinct branches:

  • Global Structure Mining (GSM): This branch analyzes the overall shape and structure of the predicted mask to determine if it resembles a plausible, cohesive object.
  • Pixel Weighting (PW): This branch utilizes the structural features extracted by the GSM to infer fine-grained, pixel-wise reliability maps.
The proposed APL architecture.

By explicitly modeling global structure, the APL framework avoids trusting "noisy" predictions that might have high pixel-wise confidence but nonsensical overall shapes.

3. The Min-Max Game

The training process operates as an adversarial game. The task predictor attempts to generate saliency masks that can fool the pace-generator, while the pace-generator continuously improves its ability to identify flaws in those masks. Over time, this dynamic ensures the model only incorporates the most "realistic" and structurally sound pseudo-labels into its training set.

Key Results and Insights

The authors evaluated their approach on four widely-used benchmark datasets: DUTS (using only 1k labels from the training split), PASCAL-S, DUT-O, and SOD. The experiments demonstrated that the FC-SOD approach is remarkably effective.

  • Superiority over Weak Supervision: APL significantly outperformed existing unsupervised, semi-supervised, and weakly supervised methods across all four datasets.
  • Competitive with Full Supervision: The most striking result is the model's efficiency. Using only 1,000 annotated images, the APL model performed comparably to—and sometimes better than—established models trained on the full 10,000+ image DUTS dataset. It achieves roughly 95% of the performance of a fully supervised model while using 90% fewer manual annotations.
  • Robustness: Ablation studies showed that the APL mechanism is much more effective at refining pseudo-labels than standard techniques like "GrabCut" or traditional SPL regularizers.
Visual comparisons of weight maps generated by different methods.

The visual comparisons highlight why APL succeeds: traditional GAN and SPL methods often produce fragmented or overly blurred weight maps, whereas APL successfully isolates the cohesive structure of the salient objects.

Limitations and Future Directions

While highly effective, the approach does have limitations. Unlike semantic segmentation, where a model looks for specific, consistent classes (like "car" or "dog"), salient objects can be virtually anything. This high intra-class variance makes the "learning ambiguity" issue particularly challenging for few-cost scenarios. Additionally, the adversarial training process introduces computational overhead and longer training times compared to standard supervised learning.

Despite these challenges, the data-driven curriculum of Adversarial-Paced Learning represents a significant step forward. The authors suggest this framework could be readily extended to other dense prediction tasks—such as medical image segmentation or edge detection—where pixel-perfect annotations are similarly expensive and difficult to obtain.

Summary updated

Abstract

Detecting and segmenting salient objects from given image scenes has received great attention in recent years. A fundamental challenge in training the existing deep saliency detection models is the requirement of large amounts of annotated data. While gathering large quantities of training data becomes cheap and easy, annotating the data is an expensive process in terms of time, labor and human expertise. To address this problem, this paper proposes to learn the effective salient object detection model based on the manual annotation on a few training images only, thus dramatically alleviating human labor in training models. To this end, we name this new task as the few-cost salient object detection and propose an adversarial-paced learning (APL)-based framework to facilitate the few-cost learning scenario. Essentially, APL is derived from the self-paced learning (SPL) regime but it infers the robust learning pace through the data-driven adversarial learning mechanism rather than the heuristic design of the learning regularizer. Comprehensive experiments on four widely-used benchmark datasets have demonstrated that the proposed approach can effectively approach to the existing supervised deep salient object detection models with only 1k human-annotated training images.
BibTeX
@inproceedings{zhang2020few,
  author = {Dingwen Zhang and Haibin Tian and Jungong Han},
  title = {{Few-Cost Salient Object Detection with Adversarial-Paced Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=rovNPYDRf_J},
}
Figure