A Deep Spatial Contextual Long-term Recurrent Convolutional Network for Saliency Detection

Nian Liu, Junwei Han

Abstract

Traditional saliency models usually adopt hand-crafted image features and human-designed mechanisms to calculate local or global contrast. In this paper, we propose a novel computational saliency model, i.e., deep spatial contextual long-term recurrent convolutional network (DSCLRCN) to predict where people looks in natural scenes. DSCLRCN first automatically learns saliency related local features on each image location in parallel. Then, in contrast with most other deep network based saliency models which infer saliency in local contexts, DSCLRCN can mimic the cortical lateral inhibition mechanisms in human visual system to incorporate global contexts to assess the saliency of each image location by leveraging the deep spatial long short-term memory (DSLSTM) model. Moreover, we also integrate scene context modulation in DSLSTM for saliency inference, leading to a novel deep spatial contextual LSTM (DSCLSTM) model. The whole network can be trained end-to-end and works efficiently when testing. Experimental results on two benchmark datasets show that DSCLRCN can achieve state-of-the-art performance on saliency detection.

Introduction

Saliency detection—predicting which parts of an image will grab a human's attention—is a fundamental challenge in computer vision. Traditionally, models focused on local contrast, such as a red dot on a green field. However, human vision is far more sophisticated, utilizing global context (comparing a region to the entire image) and scene context (e.g., recognizing that a pedestrian is more salient in a forest than on a busy city street).

While deep learning has advanced the field, many existing Convolutional Neural Networks (CNNs) suffer from limited receptive fields, primarily focusing on local neighborhoods. Furthermore, few models effectively integrate the high-level "gist" of a scene to modulate saliency. The proposed DSCLRCN bridges this gap by combining local image features with a novel "Deep Spatial Contextual LSTM" to mimic the human brain's ability to process global surroundings and high-level scene context.

Methodology: The DSCLRCN Architecture

The DSCLRCN model is a multi-stage architecture designed for end-to-end training and efficient inference.

The DSCLRCN architecture integrates local feature extraction, scene context modulation, and a multi-directional spatial LSTM.

The pipeline consists of several key components:

  1. Local Feature Extraction: A pre-trained CNN (VGG-16 or ResNet-50) is modified with dilated convolutions to extract high-resolution features without losing spatial detail through excessive downsampling.
  2. Scene Feature Extraction: Simultaneously, a separate "Places-CNN" extracts a 128-dimensional vector representing the overall scene category (e.g., "office," "forest").
  3. Deep Spatial LSTM (DSCLSTM): This core innovation processes data over space rather than time. It scans the feature map in four directions (left-to-right, right-to-left, top-to-bottom, and bottom-to-top), allowing every pixel to communicate with every other pixel. This effectively mimics the lateral inhibition found in the human visual cortex.
  4. Scene Modulation: The scene vector is injected into the spatial LSTM, providing "top-down" information that tells the model which types of objects are likely to be salient given the environment.
  5. Optimization: The network is trained using a Normalized Scanpath Saliency (NSS) loss function, directly optimizing the model to match human eye-tracking data.

Experimental Evaluation

The model was evaluated on two primary benchmarks: SALICON, the largest saliency dataset with 10,000 training images, and MIT300, a highly competitive benchmark with 300 natural images and hidden ground-truth data.

Comparison of DSCLRCN with state-of-the-art models on the MIT300 benchmark.

As shown in the performance table, DSCLRCN achieved state-of-the-art results. On the MIT300 benchmark, it reached an AUC-Judd of 0.87 and a leading NSS of 2.35, surpassing 14 contemporary models. On SALICON, it outperformed previous leaders like DeepGaze II and ML-Net, particularly in NSS (3.157) and Correlation Coefficient (0.831). Qualitatively, the model excels at handling cluttered scenes where traditional local-contrast models often produce noisy saliency maps.

Key Insights

  • Global Context: Increasing the receptive field of a standard CNN improves results, but the spatial LSTM provides a more efficient mechanism for capturing truly global dependencies.
  • Biological Inspiration: The four-way scanning of the SLSTM acts like the brain's lateral interconnections, where neurons inhibit neighbors to highlight distinctive features.
  • The "Gist" Matters: Integrating top-down scene information significantly boosts performance by helping the model ignore common "distractors" specific to certain environments.
  • Competitive Saliency: A final Softmax activation layer forces competition between pixels, ensuring only the most salient regions are highlighted.

Conclusion

DSCLRCN demonstrates that incorporating global spatial interconnections and scene context modulation can significantly improve saliency detection. While adding more than two SLSTM layers showed diminishing returns, the current architecture remains highly efficient, processing images in approximately 0.27 seconds on a GTX Titan X GPU. Future work may explore more complex top-down factors, such as task-driven attention, and apply these spatial recurrent mechanisms to other dense prediction tasks like semantic segmentation.

Summary updated

Abstract

Traditional saliency models usually adopt hand-crafted image features and human-designed mechanisms to calculate local or global contrast. In this paper, we propose a novel computational saliency model, i.e., deep spatial contextual long-term recurrent convolutional network (DSCLRCN) to predict where people looks in natural scenes. DSCLRCN first automatically learns saliency related local features on each image location in parallel. Then, in contrast with most other deep network based saliency models which infer saliency in local contexts, DSCLRCN can mimic the cortical lateral inhibition mechanisms in human visual system to incorporate global contexts to assess the saliency of each image location by leveraging the deep spatial long short-term memory (DSLSTM) model. Moreover, we also integrate scene context modulation in DSLSTM for saliency inference, leading to a novel deep spatial contextual LSTM (DSCLSTM) model. The whole network can be trained end-to-end and works efficiently when testing. Experimental results on two benchmark datasets show that DSCLRCN can achieve state-of-the-art performance on saliency detection. Furthermore, the proposed DSCLSTM model can significantly boost the saliency detection performance by incorporating both global spatial interconnections and scene context modulation, which may uncover novel inspirations for studies on them in computational saliency models.
BibTeX
@article{liu2018deep,
  author = {Nian Liu and Junwei Han},
  title = {{A Deep Spatial Contextual Long-term Recurrent Convolutional Network for Saliency Detection}},
  journal = {IEEE Trans. Image Process.},
  year = {2018},
  url = {https://openreview.net/forum?id=WR4dD8CKht},
}
Figure