Pixels, Regions, and Objects: Multiple Enhancement for Salient Object Detection

Yi Wang, Ruili Wang, Xin Fan, Tianzhu Wang, Xiangjian HeCVPR 2023 · OpenReview

Abstract

Salient object detection (SOD) aims to mimic the human visual system (HVS) and cognition mechanisms to identify and segment salient objects. However, due to the complexity of these mechanisms, current methods are not perfect. Accuracy and robustness need to be further improved, particularly in complex scenes with multiple objects and background clutter. To address this issue, we propose a novel approach called Multiple Enhancement Network (MENet) that adopts the boundary sensibility, content integrity, iterative refinement, and frequency decomposition mechanisms of HVS. A multi-level hybrid loss is firstly designed to guide the network to learn pixel-level, region-level, and object-level features. A flexible multiscale feature enhancement module (ME-Module) is then designed to gradually aggregate and refine global or detailed features by changing the size order of the input feature sequence. An iterative training strategy is used to enhance boundary features and adaptive features in the dual-branch decoder of MENet. Comprehensive evaluations on six challenging benchmark datasets show that MENet achieves state-of-the-art results.

Introduction

Salient Object Detection (SOD) is a fundamental task in computer vision that involves identifying and segmenting the most visually prominent objects in an image. While deep learning has significantly advanced the field, existing models often struggle in complex environments characterized by geometric complexity, background clutter, and the inherent tension between maintaining object integrity and capturing fine details.

Humans naturally balance boundary sensitivity with global content integrity. Current SOD methods frequently fail to replicate this balance, often producing "leaky" segmentations in noisy backgrounds or losing thin structures. To bridge this gap, we introduce the Multiple Enhancement Network (MENet), which incorporates mechanisms inspired by the Human Visual System (HVS) to improve both accuracy and robustness.

Methods

MENet utilizes a U-Net-like architecture enhanced by several novel components designed to mimic human cognitive processes.

Dual-Branch Decoder and ME-Module

Instead of a standard single-path decoder, MENet employs a two-stream approach. One branch focuses on gradients (high-frequency boundary information), while the other processes adaptive features (low-frequency internal object information). By decoupling these streams, the model prevents noisy boundary data from corrupting the global consistency of the object's body.

Central to this architecture is the Multiscale Feature Enhancement Module (ME-Module). This module utilizes Atrous Spatial Pyramid Pooling (ASPP) and attention mechanisms. Its primary innovation is its flexibility: by altering the size order of the input feature sequence, the module can be toggled to prioritize either global feature aggregation or local detail refinement.

The MENet architecture features a dual-branch decoder (Gradient and Adaptive) and the flexible ME-Module for iterative feature refinement.

Iterative Refinement Strategy

The network employs a four-round iterative training strategy, mimicking a "look closer, then step back" heuristic. Odd-numbered rounds focus on extracting global features to establish object wholeness, while even-numbered rounds refine these features with local details to sharpen boundaries.

Multi-level Hybrid Loss

To ensure the model learns at various granularities, we implement a hybrid loss function:

  1. Pixel-level: Standard Binary Cross Entropy (BCE) for point-wise accuracy.
  2. Region-level: Employs SSIM and IoU across four sub-regions to enforce structural consistency within local areas.
  3. Object-level: A specialized loss inspired by the S-measure that evaluates the contrast and distribution of the entire foreground object against the ground truth.

Datasets and Evaluation

The model was pre-trained on ImageNet and fine-tuned on DUTS-TR. We evaluated MENet across six benchmark datasets: OMRON, DUTS-TE, HKU-IS, PASCAL-S, ECSSD, and SOD. These datasets represent a wide range of challenges, including multiple scattered objects, complex backgrounds, and semantically complex structures.

Key Results

MENet demonstrates superior performance compared to 16 recent state-of-the-art methods, including ICON and EGNet.

Quantitative and Qualitative Performance

On the OMRON dataset, MENet achieved a state-of-the-art Mean Absolute Error (MAE) of 0.0450. Qualitatively, the model excels at suppressing background noise and maintaining sharp, closed boundaries, even when the salient object's appearance is highly similar to its surroundings.

Quantitative MAE comparison across datasets and qualitative visual results demonstrating MENet's superior boundary definition and noise suppression.

Efficiency

Despite its iterative nature, MENet is highly efficient, achieving a processing speed of 45 FPS on an NVIDIA A100. This makes it suitable for real-time applications where both high accuracy and speed are required.

Conclusion

MENet advances salient object detection by integrating HVS-inspired mechanisms into a deep learning framework. Through its dual-branch decoder, flexible ME-Module, and multi-level hybrid loss, it effectively balances global object integrity with local detail. While low-contrast and blurred scenes remain a challenge for future research, MENet sets a new benchmark for robustness in complex SOD tasks.

Summary updated

Abstract

Salient object detection (SOD) aims to mimic the human visual system (HVS) and cognition mechanisms to identify and segment salient objects. However, due to the complexity of these mechanisms, current methods are not perfect. Accuracy and robustness need to be further improved, particularly in complex scenes with multiple objects and background clutter. To address this issue, we propose a novel approach called Multiple Enhancement Network (MENet) that adopts the boundary sensibility, content integrity, iterative refinement, and frequency decomposition mechanisms of HVS. A multi-level hybrid loss is firstly designed to guide the network to learn pixel-level, region-level, and object-level features. A flexible multiscale feature enhancement module (ME-Module) is then designed to gradually aggregate and refine global or detailed features by changing the size order of the input feature sequence. An iterative training strategy is used to enhance boundary features and adaptive features in the dual-branch decoder of MENet. Comprehensive evaluations on six challenging benchmark datasets show that MENet achieves state-of-the-art results. Both the codes and results are publicly available at https://github.com/yiwangtz/MENet.
BibTeX
@inproceedings{wang2023pixels,
  author = {Yi Wang and Ruili Wang and Xin Fan and Tianzhu Wang and Xiangjian He},
  title = {{Pixels, Regions, and Objects: Multiple Enhancement for Salient Object Detection}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=QTjq3akxya},
}
Figure