Saliency Detection with a Deeper Investigation of Light Field

Jun Zhang, Meng Wang, Jun Gao, Yi Wang, Xudong Zhang, Xindong WuIJCAI 2015

Abstract

Saliency detection aims to identify the most visually significant regions in an image. While traditional methods rely on 2D color and intensity or 3D depth maps from RGBD sensors, this paper explores the unique advantages of light field data. We propose a novel framework that leverages focal stacks and depth maps to distinguish foreground objects from cluttered backgrounds. By integrating color-depth contrast with background priors derived from focusness, our model achieves a significant performance boost over previous state-of-the-art methods on the Light Field Saliency Dataset (LFSD).

Introduction

Traditional saliency detection often struggles in complex scenes where foreground objects share similar color or texture with the background. While 3D methods using depth information (e.g., from Kinect) offer a solution, they typically require high-quality depth maps and often overlook the relationship between appearance and focus.

Light field cameras, such as the Lytro, capture the direction of light rays, enabling post-capture refocusing and the generation of a "focal stack." This paper investigates how these unique properties—specifically the ability to isolate background slices through focus—can be used to refine saliency maps in challenging, cluttered environments. The core insight is that color and depth are complementary: depth contrast helps when color contrast is low, and vice versa.

Methodology

The proposed saliency detection model follows a multi-stage pipeline that integrates three distinct light field cues:

  1. Light Field Contrast-based Saliency: The image is segmented into superpixels using the SLIC algorithm. A contrast saliency score is calculated by combining color contrast (from the all-focus image) and depth contrast (from the light field depth map).
  2. Background Priors from Focusness: A key innovation of this work is analyzing the focal stack to identify the background "slice." By calculating a focusness map for this slice, the model estimates the probability that a region belongs to the background.
  3. Background Weighted Saliency: The contrast-based saliency is weighted by the background probability. If a region is identified as part of the background based on the focal stack, its saliency score is suppressed.
  4. Optimization: Finally, a saliency optimization algorithm is applied to produce a sharp, binary-like saliency map.
The multi-stage pipeline for light field saliency detection, integrating depth, color, and focusness cues.

Experimental Results

The study utilized the Light Field Saliency Dataset (LFSD), containing 100 light fields captured with a Lytro camera. The proposed method outperformed both traditional 2D/3D models and the previous state-of-the-art light field methods. On the LFSD dataset, the model achieved an F-measure of 0.8186 and an AUC of 0.9641, compared to the 0.7500 F-measure of the previous leading method.

ROC curves demonstrating the performance gains achieved by combining color, depth, and background (bg) priors.

Qualitatively, the model excels at handling cluttered backgrounds. While other models might highlight the entire background when textures are similar, this approach successfully suppresses non-salient regions using focusness cues. Furthermore, the authors found that existing 2D saliency models can be improved by 1–5% in F-measure simply by incorporating the proposed light field depth contrast.

Conclusion and Future Work

This research demonstrates that light field focusness and depth provide powerful cues for saliency detection that go beyond standard RGBD data. While current consumer light field cameras have limited depth relief, future improvements in depth map resolution will likely further enhance these techniques. Future work may involve using these light field cues to predict human gaze shifts in real 3D scenes or developing higher-resolution depth estimation techniques.

Summary updated

BibTeX
@inproceedings{zhang2015saliency,
  author = {Jun Zhang and Meng Wang and Jun Gao and Yi Wang and Xudong Zhang and Xindong Wu},
  title = {{Saliency Detection with a Deeper Investigation of Light Field}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
}
Figure