Abstract
SaltiNet is a deep learning architecture designed to predict human scan-paths—the sequence and duration of eye fixations—on 360-degree images. By introducing a novel 3D representation called Saliency Volumes, the model captures both the spatial and temporal aspects of visual attention. SaltiNet leverages transfer learning and specific sampling heuristics to outperform existing methods, ultimately winning the Salient360! Challenge at ICME 2017.
Introduction
Traditional saliency prediction in computer vision typically produces a static 2D "heatmap" or saliency map. While these maps identify regions likely to attract attention, they are "temporally blind," failing to account for the order of fixations or their duration. In the context of 360-degree images and immersive VR/AR environments, this temporal dimension is critical. Understanding typical scan-paths allows for optimized video encoding—streaming higher quality only where a user is likely to look next—and helps designers create more engaging experiences. However, predicting a sequence of coordinates is significantly more challenging than predicting a static map, especially given the scarcity of large-scale 360-degree datasets.
Methodology
Saliency Volumes
The core innovation of SaltiNet is the Saliency Volume. Instead of a standard 2D map, the authors represent human attention as a 3D volume where the third axis represents time. Fixation timestamps are quantized into discrete slices, and a Gaussian kernel is applied to create a "probability volume" indicating where a user might be looking at various time steps ($t=0, t=1, \dots, t=n$).
SaltiNet Architecture
SaltiNet is a 10-layer Deep Convolutional Neural Network (DCNN) that takes a 360-degree equirectangular image as input and outputs a saliency volume. The architecture is based on SalNet, with the initial layers pre-trained on VGG-16 to leverage features from millions of standard 2D images. The network includes convolutional layers, max pooling, upsampling, and a final sigmoid activation to produce the probability volume.
Scan-path Sampling
To convert the continuous probability distributions of the saliency volume into discrete scan-paths, the authors evaluated three sampling strategies:
- Naive Sampling: Selecting a point in each time slice independently.
- Limiting Distance: Constraining the next fixation to be within a certain distance of the previous one, mimicking natural saccadic eye movements.
- Area Suppression: Preventing the model from immediately re-fixating on the same location.
Datasets and Training
Due to the limited availability of 360-degree eye-tracking data, the authors employed a transfer learning pipeline:
- SALICON & iSUN: Large-scale 2D saliency datasets used for initial pre-training and refinement.
- Salient360!: The primary benchmark, consisting of 60 equirectangular images with head and eye-tracking data from approximately 40 observers.
Results
SaltiNet was the top-performing solution for scan-path prediction at the Salient360! Challenge (ICME 2017). Using the Jarodzka metric (where lower scores indicate higher similarity to human paths), SaltiNet achieved a score of 2.8697, significantly outperforming competing models from SJTU (4.6565) and Wuhan University (5.9517).
The research also highlighted the importance of movement constraints; the "Limiting distance" sampling strategy achieved a score of 2.27, compared to 3.45 for the naive approach, demonstrating that incorporating physical eye-movement constraints is vital for realistic path generation.
Conclusion
SaltiNet demonstrates that treating time as a structural dimension in a saliency volume allows standard CNNs to learn complex temporal patterns without the need for recurrent architectures like LSTMs. While the current model relies on post-hoc sampling heuristics to maintain path coherence, it sets a new benchmark for scan-path prediction in immersive environments. Future work will explore end-to-end neural architectures that can dynamically predict fixation durations and path sequences directly from image content.
Summary updated