Depth-Induced Multi-Scale Recurrent Attention Network for Saliency Detection

Yongri Piao, Wei Ji, Jingjing Li, Miao Zhang, Huchuan Lu

Abstract

Salient Object Detection (SOD) is a fundamental task in computer vision aimed at identifying the most visually prominent objects in an image. While RGB-based models have seen significant progress, they often struggle in complex environments with low contrast or cluttered backgrounds. This paper introduces DMRANet, a novel deep learning architecture that effectively integrates depth information (RGB-D) to provide structural cues. By mimicking the iterative refinement process of human visual attention, DMRANet employs a recurrent attention mechanism and depth-guided multi-scale weighting to achieve state-of-the-art performance across seven benchmark datasets.

Introduction

Traditional SOD models frequently fail in scenarios where objects are camouflaged against their backgrounds or in low-light conditions. Depth information offers a solution by providing geometric and structural context that is invariant to color and texture. However, existing RGB-D methods often suffer from two primary limitations: naive fusion strategies that ignore the multi-level relationship between RGB and depth data, and static, single-pass processing that lacks the refinement capabilities of the human brain's Internal Generative Mechanism (IGM).

DMRANet addresses these gaps through a two-stream architecture that treats depth as a structural guide rather than just an additional feature channel. The model introduces three key components: the Depth Refinement Block (DRB), the Depth-induced Multi-scale Weighting (DMSW) module, and the Recurrent Attention Module (RAM).

Proposed Methodology

The overall architecture of DMRANet, illustrating the two-stream VGG-19 backbone, multi-level feature fusion via DRBs, and the sequential DMSW and RAM modules.

The DMRANet architecture is built upon a VGG-19 backbone and processes RGB and depth inputs through parallel streams. The integration is handled by three specialized modules:

1. Depth Refinement Block (DRB)

Instead of simple concatenation, the DRB uses residual connections to learn depth-specific cues. It extracts complementary information from both streams at multiple levels, ensuring that high-level semantic information and low-level spatial details are preserved and fused effectively.

2. Depth-induced Multi-scale Weighting (DMSW)

Recognizing that objects appear at different scales depending on their distance from the camera, the DMSW module uses the depth map to generate a "depth vector." This vector acts as a scale factor to weight multi-scale context features generated via dilated convolutions. This allows the network to dynamically adjust its focus based on the spatial depth of the scene.

3. Recurrent Attention Module (RAM)

Inspired by the human brain's iterative "look and refine" process, the RAM utilizes a ConvLSTM to progressively optimize the saliency map. By maintaining a memory of previous iterations, the module can suppress background noise and sharpen object boundaries over multiple time steps.

Datasets and Evaluation

The model was evaluated on six public datasets: NJUD, NLPR, LFSD, STEREO, RGBD135, and SSD. Additionally, the authors introduced a new large-scale RGB-D dataset containing 1,200 paired images. This dataset is specifically designed to be more challenging, featuring transparent objects, multiple salient targets, and highly cluttered backgrounds.

Experimental Results

Qualitative comparison of DMRANet against 16 state-of-the-art methods across various challenging scenarios, including transparent objects and low-contrast environments.

Quantitative Performance

DMRANet outperformed 16 state-of-the-art methods, ranking first across all seven datasets. On the challenging NJUD dataset, it achieved an F-measure of 0.872, significantly surpassing previous top-performing models like PCA and PDNet. It also maintained the lowest Mean Absolute Error (MAE) across the board, indicating superior precision in boundary localization.

Qualitative Analysis

Visual results demonstrate the model's robustness. While competing models often produce blurry maps or fail to detect objects with similar colors to the background, DMRANet successfully isolates complex targets, such as transparent glass bottles and small objects in dark rooms. The recurrent loops in the RAM are particularly effective at eliminating background "irritants" that confuse single-pass models.

Conclusion

DMRANet represents a significant advancement in RGB-D saliency detection by moving beyond naive fusion to a depth-guided, iterative refinement approach. The introduction of the DMSW and RAM modules allows the network to leverage depth as a structural guide for scale and attention. Future work may focus on optimizing the computational efficiency of the recurrent modules and improving robustness to noisy depth data from low-end sensors.

Summary updated

Abstract

In this work, we propose a novel depth-induced multi-scale recurrent attention network for saliency detection. It achieves dramatic performance especially in complex scenarios. There are three main contributions of our network that are experimentally demonstrated to have significant practical merits. First, we design an effective depth refinement block using residual connections to fully extract and fuse multi-level paired complementary cues from RGB and depth streams. Second, depth cues with abundant spatial information are innovatively combined with multi-scale context features for accurately locating salient objects. Third, we boost our model's performance by a novel recurrent attention module inspired by Internal Generative Mechanism of human brain. This module can generate more accurate saliency results via comprehensively learning the internal semantic relation of the fused feature and progressively optimizing local details with memory-oriented scene understanding. In addition, we create a large scale RGB-D dataset containing more complex scenarios, which can contribute to comprehensively evaluating saliency models. Extensive experiments on six public datasets and ours demonstrate that our method can accurately identify salient objects and achieve consistently superior performance over 16 state-of-the-art RGB and RGB-D approaches.
BibTeX
@inproceedings{piao2019depth,
  author = {Yongri Piao and Wei Ji and Jingjing Li and Miao Zhang and Huchuan Lu},
  title = {{Depth-Induced Multi-Scale Recurrent Attention Network for Saliency Detection}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00735},
}
Figure