TL;DR
The researchers present DSCLRCN, a deep learning model that predicts human eye fixations by combining local image features with a novel "Deep Spatial Contextual LSTM." By mimicking the human brain's ability to process global surroundings and high-level scene "gist," the model achieves state-of-the-art performance on major saliency benchmarks.
What Problem Does This Paper Address?
Saliency detection—predicting which parts of an image will grab a human's attention—is a fundamental challenge in computer vision. Traditionally, models focused on local contrast (e.g., a red dot on a green field). However, human vision is far more sophisticated. We use global context (comparing a region to the entire image) and scene context (knowing that a pedestrian is more "salient" in a forest than on a busy city street).
While deep learning has improved this field, many existing Convolutional Neural Networks (CNNs) have a limited "receptive field," meaning they still mostly look at local neighborhoods. Furthermore, few models effectively integrate the high-level "gist" of a scene to modulate what they find important. This paper aims to bridge that gap by creating a network that "looks" at the whole image holistically.
Methods
The authors propose a multi-stage architecture called the Deep Spatial Contextual Long-term Recurrent Convolutional Network (DSCLRCN):
- Local Feature Extraction: They use a pre-trained CNN (either VGG-16 or ResNet-50) modified with dilated convolutions. This allows the network to extract high-resolution features without losing spatial detail through excessive downsampling.
- Scene Feature Extraction: Simultaneously, a separate "Places-CNN" extracts a 128-dimensional vector representing the overall scene category (e.g., "office," "forest," "street").
- Deep Spatial LSTM (DSCLSTM): This is the core innovation. Instead of processing data over time, this LSTM processes data over space. It scans the image feature map in four directions (left-to-right, right-to-left, top-to-bottom, and bottom-to-top). This allows every pixel to "communicate" with every other pixel, effectively mimicking the lateral inhibition found in the human visual cortex.
- Scene Modulation: The scene vector is injected into the first step of the spatial LSTM. This "top-down" information tells the model which types of objects are likely to be salient given the environment.
- End-to-End Training: The entire pipeline is trained using a Normalized Scanpath Saliency (NSS) loss function, which directly optimizes the model to match human eye-tracking data.
Datasets
The model was evaluated on two primary benchmarks:
- SALICON: Currently the largest saliency dataset, featuring 10,000 training images with "mouse-contingent" saliency maps (a proxy for eye-tracking).
- MIT300: A standard, highly competitive benchmark consisting of 300 natural images with hidden ground-truth data.
- MIT1003: Used as a supplementary dataset to fine-tune the model for the MIT300 benchmark.
Key Insights and Ideas
- Global Context is King: The authors found that simply increasing the receptive field of a standard CNN (FCN) improved results, but the spatial LSTM provided a much more efficient way to capture truly global dependencies.
- Mimicking Biology: The four-way scanning of the SLSTM acts like the brain's lateral interconnections, where neurons inhibit their neighbors to highlight the most distinctive features.
- The "Gist" Matters: Integrating scene-level information (top-down) significantly boosted performance. Knowing the environment helps the model ignore "distractors" that might look locally salient but are common in that specific scene.
- Softmax for Competition: Using a Softmax activation at the final layer forces a "competition" between pixels, ensuring that only the most salient regions stand out.
Key Results
DSCLRCN achieved state-of-the-art results at the time of publication:
- On SALICON: It outperformed previous leaders like DeepGaze II and ML-Net, particularly in NSS (3.157) and Correlation Coefficient (0.831).
- On MIT300: It reached an AUC-Judd of 0.87 and an NSS of 2.35, surpassing 14 other contemporary models.
- Qualitative Success: The model proved exceptionally good at handling "cluttered" scenes and images with no obvious single focal point—scenarios where traditional local-contrast models often fail by producing "noisy" saliency maps.
- Efficiency: Despite the complex recurrent structure, the model processes an image in approximately 0.27 seconds on a GTX Titan X GPU.
Limitations and Future Work
The authors noted a "diminishing returns" effect when deepening the spatial LSTM; adding more than two SLSTM layers did not yield significant improvements.
Future research directions suggested by the work include:
- Exploring more complex top-down factors beyond just scene category (e.g., specific task-driven attention).
- Investigating different methods for fusing multiscale features, as their experiments showed that while multiscale helps ResNet models, it was harder to integrate effectively into the VGG-based version.
- Applying the spatial recurrent mechanism to other "dense prediction" tasks like semantic segmentation or object detection.
Summary updated