TempSAL - Uncovering Temporal Information for Deep Saliency Prediction

Bahar Aydemir, Ludo Hoffstetter, Tong Zhang, Mathieu Salzmann, Sabine Süsstrunk

Introduction

Saliency prediction—the task of modeling which parts of an image catch a human's eye—has traditionally treated gaze as a static destination. Most models take an image and output a single "heat map" representing the sum of all fixations over a few seconds. However, human attention is dynamic. When viewing a scene, gaze follows a trajectory, moving from primary points of interest to secondary details. Existing models often lose this temporal context, missing crucial behaviors like "inhibition of return" (the tendency to look at new things once old ones are processed).

Methods

The authors introduce TempSAL, a multi-branch architecture designed to exploit temporal patterns. The model uses a shared PNASNet-5 encoder to extract image features, which are then processed by two specialized branches: a Temporal Saliency Decoder ($D_T$) predicting five distinct 1-second saliency slices ($T_1$ to $T_5$), and an Image Saliency Decoder ($D_S$) for the standard aggregate map.

Temporal saliency prediction compared to ground truth across five one-second intervals.

A core innovation is the Spatiotemporal Mixing Module (SMM). This U-Net-inspired module fuses the temporal maps, the static map, and high-level encoder features to produce a final, refined saliency map ($S_R$). This allows the model to use the predicted temporal trajectory to sharpen the final static prediction.

Key Insights

Analysis of the temporal slices revealed several patterns in human visual attention:

  • Reading Bias: In the first second ($T_1$), attention shifts significantly from left to right, likely influenced by Western reading habits.
  • Attention Dispersion: Human agreement on saliency is highest in the first second. As time progresses ($T_2$ through $T_5$), gaze becomes more scattered as individuals explore the scene.
  • Center Bias Evolution: While center bias is always present, it intensifies over time as eyes settle back toward the center after processing main objects.
Evolution of average saliency maps over time, illustrating the intensification of center bias.

Results

TempSAL was evaluated on the SALICON and CodeCharts1k datasets. For SALICON, the authors re-processed the data into "SALICON-MD" to extract temporal information. TempSAL outperformed state-of-the-art models like DeepGaze IIE, achieving a Correlation Coefficient (CC) of 0.911 and a Similarity score (SIM) of 0.800. Ablation studies confirmed that the SMM significantly improves accuracy, proving that temporal information acts as a powerful regularizer for static saliency.

Conclusion

TempSAL demonstrates that when we look at something is as important as where we look. By uncovering the temporal dimension of gaze on static images, the model achieves superior performance. Future work may extend this logic to video saliency or practical applications like automated visual appeal assessment for web design, where the order of information processing is critical.

Summary updated

Abstract

Deep saliency prediction algorithms complement the object recognition features, they typically rely on additional information such as scene context, semantic relationships, gaze direction, and object dissimilarity. However, none of these models consider the temporal nature of gaze shifts during image observation. We introduce a novel saliency prediction model that learns to output saliency maps in sequential time intervals by exploiting human temporal attention patterns. Our approach locally modulates the saliency predictions by combining the learned temporal maps. Our experiments show that our method outperforms the state-of-the-art models, including a multi-duration saliency model, on the SALICON benchmark and CodeCharts1k dataset. Our code is publicly available on GitHub 1 1 https://ivrl.github.io/Tempsal/.
BibTeX
@inproceedings{aydemir2023tempsal,
  author = {Bahar Aydemir and Ludo Hoffstetter and Tong Zhang and Mathieu Salzmann and Sabine Süsstrunk},
  title = {{TempSAL - Uncovering Temporal Information for Deep Saliency Prediction}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00625},
}
Figure