Saliency detection by forward and backward cues in deep-CNN

Nevrez Imamoglu, Chi Zhang, Wataru Shimoda, Yuming Fang, Boxin Shi

TL;DR

This paper introduces a novel deep learning approach for visual saliency detection that leverages both forward and backward passes of a Convolutional Neural Network (CNN). By combining objectness scores from both directions, their method achieves state-of-the-art performance on challenging datasets, outperforming existing top-down and bottom-up saliency models.

What Problem Does This Paper Address?

Visual attention is crucial for understanding complex scenes, guiding our focus to significant regions. While human visual attention is well-studied, computationally modeling it for computer vision tasks remains a challenge. Traditional saliency detection methods often rely on handcrafted features or require extensive pixel-level ground truth annotations, which are costly to obtain. Deep learning, particularly CNNs, has shown great promise in image recognition and segmentation, but effectively applying them to unsupervised saliency detection, especially without direct supervision on saliency maps, is an active area of research. Existing deep learning approaches often focus on either top-down (object-driven) or bottom-up (stimulus-driven) mechanisms, or require costly fully supervised training. This paper aims to bridge this gap by developing a weakly supervised top-down saliency model that effectively utilizes CNN features without relying on pixel-level ground truth.

Methods

The core of this paper is a novel top-down saliency detection model built upon the VGG-16 CNN architecture, pre-trained on the ImageNet dataset for object classification. The key innovation lies in integrating features from both the forward pass (standard CNN feature extraction) and the backward pass (using backpropagation to derive saliency cues).

Here's a breakdown of their approach:

  • Objectness Scores from CNN: Instead of relying on explicit object detection, the model leverages the object class scores from the CNN's softmax output. These scores are used as initial gradients for backpropagation.
  • Forward and Backward Cues:
    • Forward Pass: Features are extracted from specific layers (Conv3, Conv4, Conv5, and the fully-connected layer) of the VGG-16 network.
  • Backward Pass: Backpropagation is applied to these layers. Crucially, they introduce partially guided backpropagation (PG). This differs from fully guided backpropagation (FG) by selectively propagating gradients, aiming to preserve texture and produce more coherent saliency maps. Negative values are removed during this process.
  • Feature Integration: The forward and backward features from each sub-model are combined element-wise.
  • Top-Down Saliency Computation (STD): The integrated forward and backward features are weighted and combined linearly across selected layers. A center-bias weighting is applied to account for CNNs' tendency to perform poorly at image borders.
  • Bottom-Up Enhancement: To further improve saliency maps, the proposed top-down saliency (STD) is modulated by a state-of-the-art bottom-up saliency model (based on graph-based manifold ranking). This combination aims to leverage the strengths of both approaches.
  • Final Saliency Map: The combined saliency map is normalized to produce the final output.

Datasets

The paper evaluates their model on two publicly available datasets:

  • ECCV2014-RGBD: This dataset contains 1000 images and is used for evaluating RGB-D saliency.
  • HKU-IS: This dataset comprises 1447 test images (from a larger set of 4447) with varying complexity, including multiple objects and low contrast.

The authors combine images from both datasets for a total of 2447 images for their evaluation.

Key Insights and Ideas

  • Leveraging Objectness without Explicit Detection: The model demonstrates that object class scores from a pre-trained CNN can serve as effective "objectness" cues for saliency detection, even for objects not explicitly trained for. This bypasses the need for separate object detectors.
  • The Power of Partially Guided Backpropagation (PG): The introduction of PG is a significant contribution. It shows that by carefully controlling the backpropagation process, one can achieve better saliency maps (more uniform areas, intact textures) compared to fully guided backpropagation, while maintaining efficiency.
  • Synergy of Top-Down and Bottom-Up: The paper strongly advocates for the complementary nature of top-down and bottom-up saliency mechanisms. Combining their proposed top-down model with a bottom-up approach leads to substantial performance improvements.
  • Weak Supervision is Sufficient: The model achieves high accuracy without requiring pixel-level ground truth saliency maps for training, making it more practical and cost-effective.

Key Results

  • State-of-the-Art Performance: The proposed model, referred to as PGM (Proposed model with bottom-up enhancement), achieves the highest AUC scores on the ECCV2014-RGBD dataset (0.9515) and is the second-best on the HKU-IS dataset (0.9569).
  • Consistent Results: The PGM model demonstrates remarkably stable performance across both datasets, indicating its robustness to varying image properties.
  • Efficiency: Despite its deep computational structure, the proposed model achieves high accuracy with a relatively low computation time of approximately two seconds per image on a CPU. This is significantly faster than some other models, particularly MDF [12].
  • PG vs. FG: The partially guided backpropagation (PG) approach performs better than fully guided backpropagation (FG) and other weakly supervised methods like BPS.

Limitations and Future Work

  • Comparison with MDF: While the proposed PGM model performs very well, the MDF [12] model achieves the best AUC on the HKU-IS dataset. The authors note that MDF [12] is also the slowest model, taking around five minutes per image for processing.
  • Dataset Specificity: The authors speculate that the performance difference of MDF [12] between datasets might be due to specific properties of the HKU-IS training and test images.
  • Future Work: The paper implicitly suggests that further exploration of the interplay between top-down and bottom-up cues, as well as refining the backpropagation techniques, could lead to even better saliency detection models. The consistent performance of their model across datasets also points towards its potential for broader applications.

Summary updated

BibTeX
@inproceedings{imamoglu2017saliency,
  author = {Nevrez Imamoglu and Chi Zhang and Wataru Shimoda and Yuming Fang and Boxin Shi},
  title = {{Saliency detection by forward and backward cues in deep-CNN}},
  booktitle = {ICIP},
  year = {2017},
  doi = {10.1109/icip.2017.8296317},
}
Figure