Multi-Modal Weights Sharing and Hierarchical Feature Fusion for RGBD Salient Object Detection

Fen Xiao, Bin Li, Yimu Peng, Chunhong Cao, Kai Hu, Xieping GaoIEEE Access 2020 · OpenReview

Abstract

Salient object detection (SOD) aims to identify and locate the most attractive regions in an image, which has been widely used in various vision tasks. Recently, with the development of RGBD sensor technology, depth information of scenes has become available for image understanding. In this paper, we systematically investigate and evaluate how to integrate depth cues in a pre-trained deep network and learn informative features for SOD. First, we propose a CNN-based cross-modal transfer learning approach, which learns knowledge from sufficient labeled RGB salient object datasets and guides the depth domain feature extraction. Then we design a feature fusion module to fuse the complementary features in a hierarchical manner. Finally, the saliency map is obtained by integrating multi-scale information step by step. Extensive experiments on five popular RGBD benchmark datasets demonstrate that our proposed approach achieves significant improvements and outperforms state-of-the-art methods.

Introduction

Salient Object Detection (SOD) is the task of identifying the most "interesting" or "attractive" regions in an image—the parts a human would naturally look at first. While traditional methods use only RGB data, they often fail when an object has a similar color to its background or when lighting is poor.

Adding depth information (RGBD) helps solve these issues, as depth cues provide clear object boundaries regardless of color. However, two major challenges remain: data scarcity, as there are far fewer labeled RGBD datasets compared to massive RGB-only datasets, and fusion complexity, as it is difficult to determine the best way to combine "flat" color data with "spatial" depth data without losing important details.

Proposed Methodology

The proposed Pre-trained Guide Hierarchical Fusion (PGHF) network addresses these challenges through three main components:

Cross-Modal Weight Sharing

The model uses two VGG16-based branches to extract features—one for RGB and one for Depth. Crucially, these branches share the same weights. The network is first trained on a massive RGB dataset (over 20,000 images), and those learned parameters are then transferred to the depth branch. This allows the model to leverage sophisticated visual patterns learned from color images in the depth domain, effectively treating depth maps as "pseudo-images."

Hierarchical Fusion Module (HF-CNN)

Instead of combining data only at the beginning (early fusion) or the end (late fusion), the HF-CNN module integrates RGB and depth features at five different scales. This allows the model to capture both high-level semantic meaning (what the object is) and low-level spatial details (where the edges are) by maintaining large receptive fields while preserving context information from upper to lower layers.

Comparison of different fusion strategies: (a) Late Fusion and (c) Hierarchical Fusion.
Detailed architecture variants: (b) Early Fusion and (d) the proposed multi-modal fusion framework highlighting the RGB, Depth, and Fusion branches.

Hybrid Loss Function

To handle class imbalance—where the background is typically much larger than the salient object—the model utilizes a combination of Binary Cross-Entropy (BCE) loss and Dice loss during training.

Experimental Results

The PGHF model was trained on a combined set of 2,050 samples from the NJUD2K and NLPR datasets and evaluated against five popular RGBD benchmarks: NJUD2K, NLPR, SSB1000, LFSD, and RGBD135.

Key performance metrics include:

  • Accuracy: On the NJUD2K dataset, the model achieved an S-measure of 0.882 and a max F-measure of 0.884, setting a new benchmark.
  • Precision: The Mean Absolute Error (MAE) reached as low as 0.021 on the RGBD135 dataset, indicating very high precision in boundary detection.
  • Efficiency: The inference time is approximately 0.042s per image pair, making it suitable for near-real-time applications.

Conclusion

This research demonstrates that sharing weights between RGB and depth branches allows a model to bypass the need for massive labeled depth datasets. Furthermore, progressive, layer-by-layer hierarchical fusion is superior to simple concatenation for maintaining structural and semantic integrity. While the model is robust, future work will focus on improving performance in extremely complex scenes and handling noisier depth data from low-end sensors.

Summary updated

Abstract

Salient object detection (SOD) aims to identify and locate the most attractive regions in an image, which has been widely used in various vision tasks. Recent years, with the development of RGBD sensor technology, depth information of scenes becomes available for image understanding. In this paper, we systematically investigate and evaluate on how to integrate depth cues in a pre-trained deep network and learn informative features for SOD. First, we propose a CNN-based cross-modal transfer learning, which learn knowledge from sufficient labeled RGB salient object datasets and guide the depth domain feature extraction. Then we design a feature fusion module to fuse the complementary features in a hierarchical manner. At last, the final saliency map is obtained by integrating multi-scale information step by step. Extensive experiments on five popular RGBD benchmark datasets demonstrate that our proposed approach achieves significant improvements and outperforms the state-of-the-art methods.
BibTeX
@inproceedings{xiao2020multi,
  author = {Fen Xiao and Bin Li and Yimu Peng and Chunhong Cao and Kai Hu and Xieping Gao},
  title = {{Multi-Modal Weights Sharing and Hierarchical Feature Fusion for RGBD Salient Object Detection}},
  booktitle = {IEEE Access},
  year = {2020},
  url = {https://openreview.net/forum?id=gy3zoVTErY},
}
Figure