SID4VAM: A Benchmark Dataset With Synthetic Images for Visual Attention Modeling

David Berga, Xosé Ramón Fernández-Vidal, Xavier Otazu, Xosé M. PardoICCV 2019 · DOI · arXiv

Introduction

Saliency modeling aims to predict "where people look" in an image. While state-of-the-art Deep Learning (DL) models excel on real-world benchmarks like MIT300 or CAT2000, these datasets are often confounded by "top-down" factors—humans look at faces or objects because they are meaningful, not necessarily because they are visually salient in a raw, sensory sense. SID4VAM addresses this by isolating bottom-up attention: the involuntary "pop-out" effect caused by low-level features.

The SID4VAM Dataset and Generator

The researchers introduced the SID4VAM Dataset, a collection of 230 synthetic images designed to test the Human Visual System's (HVS) response to basic psychophysical patterns. The dataset includes 15 types of low-level features, such as orientation, brightness, color, and size, presented across seven levels of feature contrast ranging from "hard" to "easy."

Examples of synthetic stimuli in SID4VAM across different feature types (rows 1-5) and contrast levels (columns 1-7).

To support further research, the authors also released SIG4VAM, a tool for generating parameterized psychophysical images. By randomizing target locations, SID4VAM effectively neutralizes "center bias," a common shortcut used by models to achieve high scores on natural image datasets.

Methodology

The study evaluated 27 saliency models across five categories: Cognitive/Biological, Information-Theoretic, Probabilistic, Spectral/Fourier-based, and Machine/Deep Learning. Performance was assessed using eye-tracking data from 34 participants and metrics like the Saliency Index (SI), which measures the energy of a saliency map inside a target region versus the background.

Key Results: The Deep Learning Disconnect

The benchmark revealed a significant performance gap: DL models, the current "gold standard" for real-world images, perform poorly on basic synthetic patterns.

Performance comparison (SIM metric) across model categories. Colors represent categories: Red (Cognitive), Green (Information-Theoretic), Blue (Probabilistic), Orange (Spectral), and Cyan (Deep Learning).
  • Spectral Models as Winners: Models based on Spectral/Fourier analysis, particularly the Hypercomplex Fourier Transform (HFT), outperformed all others. These models better capture the global vs. local contrast balance that drives human pop-out effects.
  • DL Performance Lag: High-ranking DL models like SAM-VGG struggled significantly, with some metrics falling below classic non-learning models. This suggests these models rely heavily on high-level object recognition rather than fundamental low-level feature integration.
  • Psychophysical Consistency: Human performance improves linearly with feature contrast. While some classic models (e.g., AWS, CASD) mirrored this human "sensitivity curve," many DL models did not, showing they do not "see" features the same way humans do.

Conclusion

SID4VAM serves as a vital "sanity check" for the computer vision community. It highlights that success in object-rich environments does not equate to a functional simulation of the human visual system's early stages. Future work will expand the dataset to include dynamic stimuli to test temporal saliency and explore the architectural reasons for DL failures on low-level tasks.

Summary updated

BibTeX
@inproceedings{berga2019sid,
  author = {David Berga and Xosé Ramón Fernández-Vidal and Xavier Otazu and Xosé M. Pardo},
  title = {{SID4VAM: A Benchmark Dataset With Synthetic Images for Visual Attention Modeling}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00888},
}
Figure