Absolute and Relative Depth-Induced Network for RGB-D Salient Object Detection

Yuqiu Kong, He Wang, Lingwei Kong, Yang Liu, Cuili Yao, Baocai Yin2023 · DOI

TL;DR

This paper introduces a novel "Depth-Induced Network" (DIN) for RGB-D salient object detection, which effectively leverages both absolute and relative depth information. By proposing new modules (ADIM and RDIM) that integrate these depth cues with RGB features in a hierarchical and spatially aware manner, the DIN achieves state-of-the-art performance with a significantly smaller model size.

What Problem Does This Paper Address?

Salient object detection aims to identify the most visually prominent objects in an image. While RGB images provide rich semantic information, they often struggle with complex scenes featuring similar foreground and background appearances or low-intensity environments. RGB-D data, which combines RGB images with depth information, offers a powerful way to overcome these limitations by providing spatial and geometric cues. However, existing RGB-D saliency detectors often only utilize absolute depth values and fuse features in a superficial manner, neglecting the valuable structural and contrastive information present in relative depth relationships between pixels. This paper addresses the challenge of effectively integrating both absolute and relative depth information with RGB features to achieve more accurate and robust salient object detection, especially in challenging scenarios.

Methods

The core of this work is the Depth-Induced Network (DIN), a single-stream architecture designed for efficient RGB-D salient object detection. The DIN comprises two key novel components:

  • Absolute Depth-Induced Module (ADIM): This module is designed to hierarchically fuse RGB and absolute depth features in the encoding stage. Inspired by Gated Recurrent Units (GRUs), ADIM treats the multi-scale feature integration as a sequential process, allowing for iterative updates and adaptive selection of informative features from both modalities. This "in-depth" fusion helps to combine geometric and semantic information more effectively than simple concatenation.
  • Relative Depth-Induced Module (RDIM): This module operates in the decoding stage and focuses on exploiting relative depth information to capture fine-grained saliency cues. It constructs a graph on the feature maps, where nodes represent pixels and edges are weighted by relative depth differences. A spatial Graph Convolutional Network (GCN) is then used to propagate information based on both spatial and semantic affinities between pixels, allowing the network to learn detailed saliency cues and refine object boundaries.

The DIN's single-stream architecture reduces computational costs compared to traditional two-stream approaches.

Datasets

The paper evaluates the proposed DIN on six large-scale RGB-D saliency detection datasets:

  • NLPR
  • NJUD
  • STERE
  • SIP
  • LFSD
  • SSD

For training, the authors use a subset of 1485 images from NJUD and 700 images from NLPR. The remaining images from these datasets, along with the other four datasets, are used for testing.

Key Insights and Ideas

  • Relative Depth is Crucial for Fine Details: The paper strongly emphasizes that relative depth information, which captures the spatial relationship and contrast between adjacent pixels, is vital for detecting subtle saliency cues and refining object boundaries, especially in challenging scenes where absolute depth might be ambiguous.
  • Hierarchical and In-Depth Fusion Matters: Simply concatenating RGB and depth features is insufficient. The ADIM's GRU-inspired approach allows for a deeper, iterative fusion of features across multiple scales, leading to a richer representation.
  • Graph Convolutional Networks for Spatial Relationships: Leveraging GCNs with relative depth information allows the model to explicitly model local spatial structures and affinities, which is highly effective for capturing detailed saliency.
  • Single-Stream Efficiency: The DIN's single-stream design significantly reduces model complexity and computational overhead without sacrificing performance, making it more practical for deployment.

Key Results

  • State-of-the-Art Performance: The proposed DIN achieves competitive results on six challenging benchmarks, outperforming most existing RGB-D salient object detection models across various metrics (F-measure, S-measure, E-measure, MAE).
  • Superiority on Complex Datasets: The DIN shows particular strength on datasets like NJUD and LFSD, which contain semantically complex images, demonstrating its ability to extract informative cues from cluttered scenes.
  • Lightweight Model: The DIN boasts a model size of 99.55 Mb and achieves a speed of 11 FPS, which is significantly smaller than many state-of-the-art methods, making it suitable for mobile applications.
  • Ablation Studies Confirm Contributions: Experiments show that both ADIM and RDIM contribute significantly to the overall performance. The ADIM improves upon a baseline by up to 3% on F-measure, while the RDIM further refines details and improves performance by an additional 3% on F-measure. The combination of both modules (DIN) yields the best results.

Limitations and Future Work

  • Failure Cases: The paper acknowledges failure cases where the salient object has very low depth contrast with its surroundings, or where depth sensors provide inaccurate or noisy information. In such scenarios, the depth information can become detrimental, leading to prediction errors.
  • Future Work: The authors suggest improving the robustness of the ADIM and RDIM in more complex scenarios. They also aim to extend the proposed models to other cross-modal tasks and explore more applications of RGB-D salient object detection in engineering contexts.

Summary updated

BibTeX
@misc{kong2023absolute,
  author = {Yuqiu Kong and He Wang and Lingwei Kong and Yang Liu and Cuili Yao and Baocai Yin},
  title = {{Absolute and Relative Depth-Induced Network for RGB-D Salient Object Detection}},
  year = {2023},
  doi = {10.3390/s23073611},
}
Figure