Introduction
Saliency prediction—the task of modeling which parts of an image catch a human's eye—has traditionally treated gaze as a static destination. Most models take an image and output a single "heat map" representing the sum of all fixations over a few seconds. However, human attention is dynamic. When viewing a scene, gaze follows a trajectory, moving from primary points of interest to secondary details. Existing models often lose this temporal context, missing crucial behaviors like "inhibition of return" (the tendency to look at new things once old ones are processed).
Methods
The authors introduce TempSAL, a multi-branch architecture designed to exploit temporal patterns. The model uses a shared PNASNet-5 encoder to extract image features, which are then processed by two specialized branches: a Temporal Saliency Decoder ($D_T$) predicting five distinct 1-second saliency slices ($T_1$ to $T_5$), and an Image Saliency Decoder ($D_S$) for the standard aggregate map.
A core innovation is the Spatiotemporal Mixing Module (SMM). This U-Net-inspired module fuses the temporal maps, the static map, and high-level encoder features to produce a final, refined saliency map ($S_R$). This allows the model to use the predicted temporal trajectory to sharpen the final static prediction.
Key Insights
Analysis of the temporal slices revealed several patterns in human visual attention:
- Reading Bias: In the first second ($T_1$), attention shifts significantly from left to right, likely influenced by Western reading habits.
- Attention Dispersion: Human agreement on saliency is highest in the first second. As time progresses ($T_2$ through $T_5$), gaze becomes more scattered as individuals explore the scene.
- Center Bias Evolution: While center bias is always present, it intensifies over time as eyes settle back toward the center after processing main objects.
Results
TempSAL was evaluated on the SALICON and CodeCharts1k datasets. For SALICON, the authors re-processed the data into "SALICON-MD" to extract temporal information. TempSAL outperformed state-of-the-art models like DeepGaze IIE, achieving a Correlation Coefficient (CC) of 0.911 and a Similarity score (SIM) of 0.800. Ablation studies confirmed that the SMM significantly improves accuracy, proving that temporal information acts as a powerful regularizer for static saliency.
Conclusion
TempSAL demonstrates that when we look at something is as important as where we look. By uncovering the temporal dimension of gaze on static images, the model achieves superior performance. Future work may extend this logic to video saliency or practical applications like automated visual appeal assessment for web design, where the order of information processing is critical.
Summary updated