A Deep Spatial Contextual Long-Term Recurrent Convolutional Network for Saliency Detection

Nian Liu, Junwei Han

TL;DR

The researchers present DSCLRCN, a deep learning model that predicts human eye fixations by combining local image features with a novel "Deep Spatial Contextual LSTM." By mimicking the human brain's ability to process global surroundings and high-level scene "gist," the model achieves state-of-the-art performance on major saliency benchmarks.

What Problem Does This Paper Address?

Saliency detection—predicting which parts of an image will grab a human's attention—is a fundamental challenge in computer vision. Traditionally, models focused on local contrast (e.g., a red dot on a green field). However, human vision is far more sophisticated. We use global context (comparing a region to the entire image) and scene context (knowing that a pedestrian is more "salient" in a forest than on a busy city street).

While deep learning has improved this field, many existing Convolutional Neural Networks (CNNs) have a limited "receptive field," meaning they still mostly look at local neighborhoods. Furthermore, few models effectively integrate the high-level "gist" of a scene to modulate what they find important. This paper aims to bridge that gap by creating a network that "looks" at the whole image holistically.

Methods

The authors propose a multi-stage architecture called the Deep Spatial Contextual Long-term Recurrent Convolutional Network (DSCLRCN):

  1. Local Feature Extraction: They use a pre-trained CNN (either VGG-16 or ResNet-50) modified with dilated convolutions. This allows the network to extract high-resolution features without losing spatial detail through excessive downsampling.
  2. Scene Feature Extraction: Simultaneously, a separate "Places-CNN" extracts a 128-dimensional vector representing the overall scene category (e.g., "office," "forest," "street").
  3. Deep Spatial LSTM (DSCLSTM): This is the core innovation. Instead of processing data over time, this LSTM processes data over space. It scans the image feature map in four directions (left-to-right, right-to-left, top-to-bottom, and bottom-to-top). This allows every pixel to "communicate" with every other pixel, effectively mimicking the lateral inhibition found in the human visual cortex.
  4. Scene Modulation: The scene vector is injected into the first step of the spatial LSTM. This "top-down" information tells the model which types of objects are likely to be salient given the environment.
  5. End-to-End Training: The entire pipeline is trained using a Normalized Scanpath Saliency (NSS) loss function, which directly optimizes the model to match human eye-tracking data.

Datasets

The model was evaluated on two primary benchmarks:

  • SALICON: Currently the largest saliency dataset, featuring 10,000 training images with "mouse-contingent" saliency maps (a proxy for eye-tracking).
  • MIT300: A standard, highly competitive benchmark consisting of 300 natural images with hidden ground-truth data.
  • MIT1003: Used as a supplementary dataset to fine-tune the model for the MIT300 benchmark.

Key Insights and Ideas

  • Global Context is King: The authors found that simply increasing the receptive field of a standard CNN (FCN) improved results, but the spatial LSTM provided a much more efficient way to capture truly global dependencies.
  • Mimicking Biology: The four-way scanning of the SLSTM acts like the brain's lateral interconnections, where neurons inhibit their neighbors to highlight the most distinctive features.
  • The "Gist" Matters: Integrating scene-level information (top-down) significantly boosted performance. Knowing the environment helps the model ignore "distractors" that might look locally salient but are common in that specific scene.
  • Softmax for Competition: Using a Softmax activation at the final layer forces a "competition" between pixels, ensuring that only the most salient regions stand out.

Key Results

DSCLRCN achieved state-of-the-art results at the time of publication:

  • On SALICON: It outperformed previous leaders like DeepGaze II and ML-Net, particularly in NSS (3.157) and Correlation Coefficient (0.831).
  • On MIT300: It reached an AUC-Judd of 0.87 and an NSS of 2.35, surpassing 14 other contemporary models.
  • Qualitative Success: The model proved exceptionally good at handling "cluttered" scenes and images with no obvious single focal point—scenarios where traditional local-contrast models often fail by producing "noisy" saliency maps.
  • Efficiency: Despite the complex recurrent structure, the model processes an image in approximately 0.27 seconds on a GTX Titan X GPU.

Limitations and Future Work

The authors noted a "diminishing returns" effect when deepening the spatial LSTM; adding more than two SLSTM layers did not yield significant improvements.

Future research directions suggested by the work include:

  • Exploring more complex top-down factors beyond just scene category (e.g., specific task-driven attention).
  • Investigating different methods for fusing multiscale features, as their experiments showed that while multiscale helps ResNet models, it was harder to integrate effectively into the VGG-based version.
  • Applying the spatial recurrent mechanism to other "dense prediction" tasks like semantic segmentation or object detection.

Summary updated

Abstract

Traditional saliency models usually adopt hand-crafted image features and human-designed mechanisms to calculate local or global contrast. In this paper, we propose a novel computational saliency model, i.e., deep spatial contextual long-term recurrent convolutional network (DSCLRCN), to predict where people look in natural scenes. DSCLRCN first automatically learns saliency related local features on each image location in parallel. Then, in contrast with most other deep network based saliency models which infer saliency in local contexts, DSCLRCN can mimic the cortical lateral inhibition mechanisms in human visual system to incorporate global contexts to assess the saliency of each image location by leveraging the deep spatial long short-term memory (DSLSTM) model. Moreover, we also integrate scene context modulation in DSLSTM for saliency inference, leading to a novel deep spatial contextual LSTM (DSCLSTM) model. The whole network can be trained end-to-end and works efficiently when testing. Experimental results on two benchmark datasets show that DSCLRCN can achieve state-of-the-art performance on saliency detection. Furthermore, the proposed DSCLSTM model can significantly boost the saliency detection performance by incorporating both global spatial interconnections and scene context modulation, which may uncover novel inspirations for studies on them in computational saliency models.
BibTeX
@article{liu2018deep,
  author = {Nian Liu and Junwei Han},
  title = {{A Deep Spatial Contextual Long-Term Recurrent Convolutional Network for Saliency Detection}},
  journal = {IEEE Trans. Image Process.},
  year = {2018},
  url = {https://openreview.net/forum?id=WR4dD8CKht},
}

Loading related content...

Figure