Abstract
Numerous efforts have been made to design different low level saliency cues for the RGBD saliency detection, such as color or depth contrast features, background and color compactness priors. However, how these saliency cues interact with each other and how to incorporate these low level saliency cues effectively to generate a master saliency map remain a challenging problem. In this paper, we design a new convolutional neural network (CNN) to fuse different low level saliency cues into hierarchical features for automatically detecting salient objects in RGBD images. In contrast to the existing works that directly feed raw image pixels to the CNN, the proposed method takes advantage of the knowledge in traditional saliency detection by adopting various meaningful and well-designed saliency feature vectors as input. This can guide the training of CNN towards detecting salient object more effectively due to the reduced learning ambiguity. We then integrate a Laplacian propagation framework with the learned CNN to extract a spatially consistent saliency map by exploiting the intrinsic structure of the input image. Extensive quantitative and qualitative experimental evaluations on three datasets demonstrate that the proposed method consistently outperforms state-of-the-art methods.
Introduction
Saliency detection aims to predict which parts of an image will grab a human's attention. While RGB-based methods are mature, adding a depth channel (RGBD) introduces a new challenge: how do you effectively combine color and depth? Existing methods often use simple heuristics or linear combinations to merge separate color and depth saliency maps. However, if one map is noisy or incorrect—for instance, if a background object has high color contrast but is physically far away—these simple methods fail.
The authors argue that we shouldn't just merge the results of saliency detection; we should use deep learning to learn the complex interactions between raw color and depth features.
Proposed Methodology
The proposed framework consists of three main modules:
- Saliency Feature Extraction: The image is first segmented into superpixels. For each superpixel, the authors calculate seven types of feature vectors: local/global color contrast, local/global depth contrast, background priors (color and depth), and color compactness.
- CNN-based Deep Fusion: Instead of feeding raw pixels into a CNN, the authors feed these pre-calculated feature vectors. These vectors are reshaped into a $32 \times 32 \times 6$ volume. The CNN, consisting of three convolutional layers and a fully connected layer, acts as a non-linear integrator, learning how to weight and combine these cues to produce a saliency probability for each superpixel.
- Laplacian Propagation: CNNs can sometimes produce spatially inconsistent results. To address this, the authors apply a Laplacian propagation framework. This technique treats the image as a manifold and ensures that neighboring superpixels with similar appearances and depths have similar saliency values, resulting in a smoother and more accurate final map.
Experimental Results
The "Deep Fusion" method was evaluated on three major RGBD benchmarks: the NLPR RGBD dataset, the NJUDS2000 dataset, and the Light Field Saliency Dataset (LFSD).
The method consistently outperformed previous state-of-the-art methods across all datasets. On the NLPR dataset, the proposed method achieved an F-measure of 0.7823, significantly higher than the next best method (GP at 0.7184). Qualitative results show the model excels in "tough" cases, such as when a salient object has a very similar color to the background but is physically closer to the camera.
The Laplacian propagation step was shown to be a "universal" booster; applying it to other existing methods also improved their performance, though the authors' CNN-based input remained the strongest.
Conclusion
The "Deep Fusion" framework demonstrates that feeding hand-designed saliency features into a CNN is more effective than feeding raw pixels for RGBD saliency detection. This approach reduces learning ambiguity by guiding the network with domain knowledge. While the model is robust, it still faces challenges when a salient object shares both similar color and similar depth information with the background. Future work may explore additional saliency cues and extend this feature-level fusion architecture to other computer vision tasks.
Summary updated