Examining visual saliency prediction in naturalistic scenes

Shafin Rahman, Mrigank Rochan, Yang Wang, Neil D. B. BruceICIP 2014 · OpenReview

Abstract: Visual saliency prediction is a growing field with numerous applications, yet most models are evaluated on urban datasets containing clear, salient objects. This study evaluates top-performing algorithms on naturalistic scenes devoid of obvious subjects. We introduce and test a biologically-inspired model based on early visual processing in primates, finding that it performs competitively with complex engineered models in natural environments. Our results highlight significant performance gaps between standard and natural datasets and identify image statistics that predict human gaze agreement.

The Challenge of Naturalistic Saliency

In computer vision, "saliency" refers to predicting where a human will look first. While many models excel on benchmarks like the Bruce & Tsotsos dataset, these often feature "composed" images with clear focal points, such as a red stop sign or a person. In these contexts, high-level cognitive priors—like looking for faces—dominate.

This research investigates how these models perform in truly naturalistic settings, such as forests or fields, where visual textures and patterns drive attention rather than distinct objects.

Methods: The Second Order Contrast (SOC) Model

The researchers benchmarked 10 leading saliency algorithms against a model inspired by the primary visual cortex (V1) of primates. This Second Order Contrast (SOC) model utilizes a three-stage pipeline:

  1. Spectral Energy: Log-Gabor filters mimic neural responses to various orientations and scales.
  2. Divisive Normalization (DN): Neural responses are suppressed by neighboring activity to prevent signal saturation in high-contrast regions.
  3. Second Order Contrast: Spatial variance of the energy is computed to identify regions that stand out from their surroundings.
The three-stage pipeline of the Second Order Contrast (SOC) model, showing spectral energy, divisive normalization, and final saliency prediction.

Evaluation was conducted using Shuffled AUC (sAUC) to account for "center bias"—the human tendency to look at the middle of an image regardless of content.

Datasets

Two primary datasets were utilized to compare performance across environments:

  • DOVES: 101 grayscale naturalistic images (e.g., forests, textures) with 30,000 human fixation points. This dataset lacks obvious salient objects.
  • Bruce & Tsotsos: A baseline of 120 color urban scenes containing clear objects.

Key Insights and Results

The study revealed a "natural penalty": saliency prediction is significantly harder in the wild. Performance on the DOVES dataset was markedly lower than on standard object-heavy datasets across all models.

  • Simplicity vs. Complexity: The SOC model achieved a Shuffled AUC of 0.556 on DOVES, tying for the highest score. This suggests that in the absence of clear objects, low-level energy and contrast are the primary drivers of attention, making complex "object-detecting" logic less effective.
  • Inter-Observer Agreement: Human observers agree far less on where to look in natural scenes (~0.55 agreement score) compared to urban scenes (~0.74).
  • Predictive Statistics: The authors found that the kurtosis of DCT coefficients—a measure of image data sparsity—can predict the level of agreement among human observers regarding a scene's focal points.

Conclusion and Future Directions

The SOC model's success on naturalistic data demonstrates the power of biologically-inspired principles. However, its current grayscale-only implementation is a limitation for color-rich benchmarks. Future work should integrate these low-level statistical properties into more robust models and expand naturalistic benchmarking to prevent algorithmic over-fitting to urban photography.

Summary updated

Abstract

Given the significant number of potential applications, visual saliency has increasingly become an area of interest in image and vision research. Many different strategies for predicting visual saliency have been proposed, that differ in their composition or rationale, and with a significant focus on improving performance across standard benchmarks. Recent benchmarks considering a large number of algorithms have further provided an understanding of the behavior of different algorithms. Performance evaluation has primarily focused on indoor and outdoor images of urban environments, many of which are composed, and contain salient objects. In this work, we test the performance of a number of the better performing algorithms on data derived from naturalistic scenes. In addition, given the strong connection to human vision, we test a putative model for early visual processing in primates tied to spectral energy and normalization. Results demonstrate significant differences between common datasets, and natural images. Performance analysis of the second-order contrast model also provides additional insight concerning the role of spectral energy in determining saliency. Finally we include analysis that demonstrates statistical properties of images that tend to imply common gaze patterns across observers.
BibTeX
@inproceedings{rahman2014examining,
  author = {Shafin Rahman and Mrigank Rochan and Yang Wang and Neil D. B. Bruce},
  title = {{Examining visual saliency prediction in naturalistic scenes}},
  booktitle = {ICIP},
  year = {2014},
  url = {https://openreview.net/forum?id=_sQvrC7kE4},
}
Figure