Improving saliency models' predictions of the next fixation with humans' intrinsic cost of gaze shifts

Florian Kadner, Tobias Thomas, David Hoppe, Constantin A. Rothkopf

TL;DR

This paper tackles the persistent gap in predicting human gaze behavior by proposing a novel sequential decision-making framework. It integrates arbitrary saliency maps with empirically measured human costs for gaze shifts and a history-dependent exploration bonus, significantly outperforming existing models in predicting the next fixation.

What Problem Does This Paper Address?

Predicting where humans will look next in an image, known as gaze prediction or visual saliency modeling, has seen significant progress. However, a substantial gap remains between current computational models and actual human eye movements. Traditional saliency models often treat images statically, producing a single "saliency map" that indicates regions of interest. Scanpath models attempt to predict sequences of fixations but struggle with the inherent variability between observers. This paper addresses this by aiming to bridge the gap between static saliency maps and dynamic, human-like gaze sequences, acknowledging that human eye movements are influenced not just by image content but also by the intrinsic costs and history of previous movements.

Methods

The core contribution is a sequential decision-making algorithm that transforms static saliency maps into dynamic, history-dependent "value maps." This is achieved by combining three key components:

  1. Saliency Map: An input from any existing saliency model, representing the image-based reward.
  2. Human Cost Function: This incorporates recently measured, empirically derived costs associated with making eye movements (saccades). These costs are independent of image content and capture preferences in movement magnitude, direction, and relative angle.
  3. Sequential Exploration Bonus: This component accounts for the history of gaze shifts. It dynamically changes with each subsequent gaze, encouraging exploration of previously unvisited or long-ignored areas. The parameters governing its spatial extent and temporal decay are learned from human gaze data.

The algorithm frames gaze prediction as a Markov Decision Process (MDP) and uses a one-step look-ahead greedy approximation to select the next fixation that maximizes the combined value. The relative contributions of these three components are optimized using the Normalized Scanpath Saliency (NSS) metric.

Datasets

  • MIT1003: Used for optimizing the model's parameters (weights for the three components) and for estimating exploration parameters.
  • OSIE: Used for evaluation.
  • Toronto Dataset: Used for evaluation.

The paper also references the MIT/Tuebingen saliency benchmark for selecting baseline saliency models.

Key Insights and Ideas

  • Human Gaze is More Than Just Saliency: The research strongly suggests that human eye movements are not solely driven by image features (saliency) but are significantly influenced by the intrinsic costs of making eye movements and the history of where the observer has looked.
  • Empirical Cost Measurement is Crucial: The ability to incorporate empirically measured, content-independent costs of gaze shifts is a key enabler for improving prediction accuracy. This moves beyond inferring these costs indirectly from image features.
  • Exploration vs. Exploitation in Gaze: The "sequential exploration bonus" effectively models the trade-off between exploiting salient regions and exploring new areas, a fundamental concept in sequential decision-making.
  • Dynamic Value Maps are Key: The core innovation is transforming static saliency maps into dynamic value maps that update after each fixation, reflecting the evolving state of the observer's interaction with the scene.

Key Results

  • The proposed algorithm, when applied to five state-of-the-art saliency models, significantly outperforms their baselines on the MIT1003, OSIE, and Toronto datasets.
  • Improvements were observed for both one-step ahead predictions (predicting the very next fixation) and n-step ahead predictions (predicting further fixations).
  • The model achieved higher NSS scores across all datasets and models, indicating better prediction of fixation locations.
  • The AUC score, a metric not used during optimization, also showed improvements, further validating the model's generalization.
  • The analysis of the exploration parameters revealed that the attractiveness of previously visited locations increases approximately linearly over time, suggesting a form of "positive recency" effect.

Limitations and Future Work

  • Computational Cost: While the paper mentions reducing image dimensions to manage computational cost, the sequential nature and recomputation of value maps after each gaze shift can still be computationally intensive.
  • Generalizability of Exploration Parameters: While the paper attempts to derive general weights for the exploration map by averaging across models, the optimal parameters might still vary depending on the specific saliency model and dataset.
  • Future Work: The authors suggest that their framework can be applied to arbitrary saliency models. Further exploration of how task-specific information can be integrated into this framework could also be a fruitful direction. The paper also implicitly points towards the need for more sophisticated scanpath similarity metrics that better capture the nuances of human gaze behavior.

Summary updated

Abstract

The human prioritization of image regions can be modeled in a time invariant fashion with saliency maps or sequentially with scanpath models. However, while both types of models have steadily improved on several benchmarks and datasets, there is still a considerable gap in predicting human gaze. Here, we leverage two recent developments to reduce this gap: theoretical analyses establishing a principled framework for predicting the next gaze target and the empirical measurement of the human cost for gaze switches independently of image content. We introduce an algorithm in the framework of sequential decision making, which converts any static saliency map into a sequence of dynamic history-dependent value maps, which are recomputed after each gaze shift. These maps are based on 1) a saliency map provided by an arbitrary saliency model, 2) the recently measured human cost function quantifying preferences in magnitude and direction of eye movements, and 3) a sequential exploration bonus, which changes with each subsequent gaze shift. The parameters of the spatial extent and temporal decay of this exploration bonus are estimated from human gaze data. The relative contributions of these three components were optimized on the MIT1003 dataset for the NSS score and are sufficient to significantly outperform predictions of the next gaze target on NSS and AUC scores for five state of the art saliency models on three image data sets.
BibTeX
@inproceedings{kadner2023improving,
  author = {Florian Kadner and Tobias Thomas and David Hoppe and Constantin A. Rothkopf},
  title = {{Improving saliency models' predictions of the next fixation with humans' intrinsic cost of gaze shifts}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=C234hMvFWQ},
}
Figure