Saliency Detection by Forward and Backward Cues in Deep-CNNs

Nevrez Imamoglu, Chi Zhang, Wataru Shimoda, Yuming Fang, Boxin Shi

Abstract

This paper presents a novel saliency detection framework that leverages both forward activations and backward gradients from a pre-trained VGG-16 network. By introducing "partially-guided" backpropagation and utilizing global objectness scores from all 1000 ImageNet classes, the proposed model (PGM) generates high-quality, uniform saliency maps. The approach achieves state-of-the-art performance on benchmark datasets while operating significantly faster than existing supervised methods, all without requiring pixel-level ground truth for training.

Introduction

Saliency detection is a fundamental task in computer vision aimed at identifying the most informative regions within an image. While deep learning has significantly advanced this field, most high-performing models rely on fully supervised learning, which necessitates expensive pixel-level annotations. Conversely, weakly supervised methods often utilize classification networks but frequently produce sparse or fragmented saliency maps. This research addresses these limitations by proposing a model that is inherently "object-aware" through the fusion of forward and backward cues, eliminating the need for saliency-specific training data.

Methodology

The Proposed Guided Model (PGM) is built upon the VGG-16 architecture. It introduces three primary innovations to improve the coherence and accuracy of saliency maps:

  1. Global Objectness Scores: Traditional backpropagation for visualization typically isolates a single target class. In contrast, PGM uses the softmax scores of all 1000 ImageNet classes as the initial gradient. This "wisdom of the crowd" approach allows the model to identify salient regions even when the primary object is not the top-predicted class.
  2. Partially-Guided Backpropagation (PG): Standard Guided Backpropagation (GBP) suppresses negative gradients at every ReLU layer, which often results in fragmented visual textures. The authors propose applying this suppression only between major sub-modules (at max-pooling layers), preserving the structural connectivity of the detected objects.
  3. Forward-Backward Fusion: The model integrates forward feature maps (representing spatial activations) with backward saliency cues (representing importance) across multiple scales (Conv3 through FC layers).
Overview of the proposed saliency detection framework combining top-down and bottom-up cues.

Finally, these top-down CNN cues are combined with a traditional bottom-up model—Graph-based Manifold Ranking—to ensure that low-level features like contrast and boundary information are preserved.

Experimental Setup and Results

The model was evaluated on the ECCV2014-RGBD and HKU-IS datasets, which contain complex scenes and multiple objects.

Quantitative Performance

The PGM model demonstrated exceptional accuracy, achieving a leading Area Under Curve (AUC) of 0.9515 on the ECCV2014-RGBD dataset. As shown in the results comparison, the model consistently outperforms or matches the performance of fully supervised models like MDF across different benchmarks.

Comparison of Area Under Curve (AUC) results across multiple benchmarks, highlighting the competitive performance of the PGM model.

Computational Efficiency

A standout feature of the PGM model is its efficiency. While the top-performing supervised competitor (MDF) requires approximately 290 seconds to process a single image on a CPU, the PGM model completes the same task in just 2 seconds. This 145x speedup makes the approach far more viable for practical applications where computational resources or time are constrained.

Conclusion

By rethinking how gradients are propagated and combined with forward features, this research demonstrates that highly accurate saliency detection is possible without expensive supervision. The discovery that partial guidance preserves structural context better than full guidance is a significant contribution to the understanding of CNN visualizations. Future work may focus on optimizing the model for real-time GPU performance and automating the selection of weighting parameters for feature fusion.

Summary updated

BibTeX
@inproceedings{imamoglu2017saliency,
  author = {Nevrez Imamoglu and Chi Zhang and Wataru Shimoda and Yuming Fang and Boxin Shi},
  title = {{Saliency Detection by Forward and Backward Cues in Deep-CNNs}},
  booktitle = {ICIP},
  year = {2017},
  doi = {10.1109/icip.2017.8296317},
}
Figure