SaltiNet: Scan-Path Prediction on 360 Degree Images Using Saliency Volumes

Marc Assens, Xavier Giró-i-Nieto, Kevin McGuinness, Noel E. O'ConnorWorkshops @ ICCV 2017 · DOI · arXiv

Abstract

SaltiNet is a deep learning architecture designed to predict human scan-paths—the sequence and duration of eye fixations—on 360-degree images. By introducing a novel 3D representation called Saliency Volumes, the model captures both the spatial and temporal aspects of visual attention. SaltiNet leverages transfer learning and specific sampling heuristics to outperform existing methods, ultimately winning the Salient360! Challenge at ICME 2017.

Introduction

Traditional saliency prediction in computer vision typically produces a static 2D "heatmap" or saliency map. While these maps identify regions likely to attract attention, they are "temporally blind," failing to account for the order of fixations or their duration. In the context of 360-degree images and immersive VR/AR environments, this temporal dimension is critical. Understanding typical scan-paths allows for optimized video encoding—streaming higher quality only where a user is likely to look next—and helps designers create more engaging experiences. However, predicting a sequence of coordinates is significantly more challenging than predicting a static map, especially given the scarcity of large-scale 360-degree datasets.

Methodology

Saliency Volumes

The core innovation of SaltiNet is the Saliency Volume. Instead of a standard 2D map, the authors represent human attention as a 3D volume where the third axis represents time. Fixation timestamps are quantized into discrete slices, and a Gaussian kernel is applied to create a "probability volume" indicating where a user might be looking at various time steps ($t=0, t=1, \dots, t=n$).

The Saliency Volume can be processed via sampling to generate scan-paths, or via addition to create traditional or temporally weighted saliency maps.

SaltiNet Architecture

SaltiNet is a 10-layer Deep Convolutional Neural Network (DCNN) that takes a 360-degree equirectangular image as input and outputs a saliency volume. The architecture is based on SalNet, with the initial layers pre-trained on VGG-16 to leverage features from millions of standard 2D images. The network includes convolutional layers, max pooling, upsampling, and a final sigmoid activation to produce the probability volume.

The SaltiNet architecture processes an equirectangular image through a series of convolutional and pooling layers to output a Saliency Volume, which is then sampled to produce scan-paths.

Scan-path Sampling

To convert the continuous probability distributions of the saliency volume into discrete scan-paths, the authors evaluated three sampling strategies:

  1. Naive Sampling: Selecting a point in each time slice independently.
  2. Limiting Distance: Constraining the next fixation to be within a certain distance of the previous one, mimicking natural saccadic eye movements.
  3. Area Suppression: Preventing the model from immediately re-fixating on the same location.

Datasets and Training

Due to the limited availability of 360-degree eye-tracking data, the authors employed a transfer learning pipeline:

  • SALICON & iSUN: Large-scale 2D saliency datasets used for initial pre-training and refinement.
  • Salient360!: The primary benchmark, consisting of 60 equirectangular images with head and eye-tracking data from approximately 40 observers.

Results

SaltiNet was the top-performing solution for scan-path prediction at the Salient360! Challenge (ICME 2017). Using the Jarodzka metric (where lower scores indicate higher similarity to human paths), SaltiNet achieved a score of 2.8697, significantly outperforming competing models from SJTU (4.6565) and Wuhan University (5.9517).

The research also highlighted the importance of movement constraints; the "Limiting distance" sampling strategy achieved a score of 2.27, compared to 3.45 for the naive approach, demonstrating that incorporating physical eye-movement constraints is vital for realistic path generation.

Conclusion

SaltiNet demonstrates that treating time as a structural dimension in a saliency volume allows standard CNNs to learn complex temporal patterns without the need for recurrent architectures like LSTMs. While the current model relies on post-hoc sampling heuristics to maintain path coherence, it sets a new benchmark for scan-path prediction in immersive environments. Future work will explore end-to-end neural architectures that can dynamically predict fixation durations and path sequences directly from image content.

Summary updated

BibTeX
@inproceedings{assens2017saltinet,
  author = {Marc Assens and Xavier Giró-i-Nieto and Kevin McGuinness and Noel E. O'Connor},
  title = {{SaltiNet: Scan-Path Prediction on 360 Degree Images Using Saliency Volumes}},
  booktitle = {Workshops @ ICCV},
  year = {2017},
  doi = {10.1109/iccvw.2017.275},
}
Figure