TL;DR
This paper introduces a "Deep Fusion" framework for RGBD saliency detection that combines traditional hand-crafted features with the learning power of Convolutional Neural Networks (CNNs). By feeding well-designed saliency cues (like color contrast and depth priors) into a CNN and refining the output with a Laplacian propagation framework, the authors achieve state-of-the-art performance in identifying the most "interesting" objects in a scene.
What Problem Does This Paper Address?
Saliency detection aims to predict which parts of an image will grab a human's attention. While RGB-based methods are mature, adding a depth channel (RGBD) introduces a new challenge: how do you effectively combine color and depth?
Existing methods often use simple heuristics or linear combinations to merge separate color and depth saliency maps. However, if one map is noisy or incorrect (e.g., a background object has high color contrast but is far away), these simple methods fail. The authors argue that we shouldn't just merge the results of saliency detection; we should use deep learning to learn the complex interactions between raw color and depth features.
Methods
The proposed framework consists of three main modules:
- Saliency Feature Extraction: The image is first segmented into superpixels. For each superpixel, the authors calculate seven types of feature vectors: local/global color contrast, local/global depth contrast, background priors (color and depth), and color compactness.
- CNN-based Deep Fusion: Instead of feeding raw pixels into a CNN, the authors feed these pre-calculated feature vectors. These vectors are reshaped into a $32 \times 32 \times 6$ volume. The CNN (consisting of three convolutional layers and a fully connected layer) acts as a non-linear integrator, learning how to weight and combine these cues to produce a saliency probability for each superpixel.
- Laplacian Propagation: CNNs can sometimes produce "noisy" or spatially inconsistent results. To fix this, the authors apply a Laplacian propagation framework. This technique treats the image as a manifold and ensures that neighboring superpixels with similar appearances and depths have similar saliency values, resulting in a much smoother and more accurate final map.
Datasets
The model was evaluated on three major RGBD benchmarks:
- NLPR RGBD dataset: 1,000 images captured via Microsoft Kinect.
- NJUDS2000 dataset: 2,000 stereo images with depth maps generated via optical flow.
- LFSD (Light Field Saliency Dataset): 100 images captured with a Lytro light field camera.
The authors used 750 images from NLPR and 1,000 from NJUDS2000 for training, using the rest for testing to demonstrate the model's ability to generalize.
Key Insights and Ideas
- Feature Vectors > Raw Pixels: A key "aha" moment is that feeding hand-designed saliency features into a CNN is more effective than feeding raw pixels for this specific task. It reduces "learning ambiguity" by guiding the network with domain knowledge (e.g., "contrast matters").
- Handling "Bad" Cues: The deep fusion approach is robust. If the depth map is low-quality (common in stereo-generated maps), the CNN learns to rely more on color cues, and vice versa.
- Global vs. Local: By including both global and local contrast vectors, the network can distinguish between an object that is locally distinct but part of a larger background pattern and an object that is truly unique in the entire scene.
Key Results
The "Deep Fusion" method consistently outperformed previous state-of-the-art methods (like LMH, ACSD, and GP) across all datasets.
- F-measure Improvements: On the NLPR dataset, the proposed method achieved an F-measure of 0.7823, significantly higher than the next best method (GP at 0.7184).
- Robustness: Qualitative results show the model excels in "tough" cases, such as when a salient object has a very similar color to the background but is physically closer to the camera.
- Refinement Power: The Laplacian propagation step was shown to be a "universal" booster; applying it to other existing methods also improved their performance, though the authors' CNN-based input remained the strongest.
Limitations and Future Work
The authors identify a primary failure case: when a salient object shares both similar color and similar depth information with the background. In these scenarios, the low-level cues are insufficient for the network to make a distinction.
For future work, the authors suggest exploring even more saliency cues (like "flash cues" from photography) and extending this "feature-level fusion" architecture to other computer vision tasks beyond saliency detection.
Summary updated