TL;DR
Researchers have developed DeepGaze III, a deep learning model that accurately predicts human eye movements (scanpaths) during free viewing of images. By combining image content with past fixation history, DeepGaze III sets a new state-of-the-art in predicting scanpaths and offers insights into the interplay of visual information and observer behavior.
What Problem Does This Paper Address?
When we look at an image, our eyes don't scan it uniformly. Instead, they move in a series of "fixations" (where we pause to gather information) linked by rapid "saccades." These sequences, called scanpaths, are not random; they are influenced by both the image content and the observer's own history of where they've looked. Predicting these scanpaths is crucial for understanding visual attention and has applications in areas like user interface design and robotics.
Previous models, often called "saliency models," focused on predicting where people might look based on image features (like edges or colors). However, these models largely ignored the sequential nature of eye movements – how one fixation influences the next. While some "scanpath models" tried to incorporate this history, they often struggled to match the performance of even basic saliency models. The challenge was to build a model that could effectively integrate both image content and the rich temporal information of a scanpath to achieve high prediction accuracy and provide deeper scientific insights.
Methods
DeepGaze III is a deep learning model built upon a modular architecture. It takes an image and a history of previous fixations as input.
- Spatial Priority Network: This part processes the image using a pre-trained DenseNet (a powerful convolutional neural network) to extract deep visual features. These features are then used to generate a "spatial priority map," which highlights image regions that are likely to attract attention, similar to traditional saliency maps.
- Scanpath Network: This component processes the history of previous fixations. It encodes the spatial relationships (distance, x/y differences) of the last few fixations.
- Fixation Selection Network: This network combines the spatial priority map and the processed scanpath history. It then outputs a conditional probability distribution, predicting the likelihood of the next fixation at any given pixel location.
- Probabilistic Generative Modeling: DeepGaze III uses a probabilistic approach, modeling the scanpath as a sequence of conditional probabilities: P(fixation_n | fixation_1...n-1, Image). This allows it to generate realistic scanpaths by sampling from these distributions.
The model's architecture allows for ablation studies, meaning researchers can systematically remove or alter components to understand their individual contributions.
Datasets
- MIT1003: Used for training and evaluating the primary model. It contains scanpath data from 15 subjects viewing 1,003 natural scenes.
- SALICON: Used for pre-training the spatial priority network.
- MIT300: A hold-out test set from the MIT/Tuebingen Saliency Benchmark, used to test model generalization to new images.
- CAT2000: Used for additional evaluation, consisting of 2,000 images across various categories, with scanpath data from 12 observers.
Key Insights and Ideas
- Scanpath History Matters, But Scene Content Dominates: For free viewing on the MIT1003 dataset, scene content has a stronger influence on fixation selection than previous scanpath history, though both are important.
- Recent History is Key: The most significant impact of scanpath history comes from the current fixation position, with earlier fixations having diminishing influence. This suggests that mechanisms like inhibition of return might decay rapidly.
- Deep Learning as a Scientific Tool: Deep learning models, due to their flexibility and capacity, can serve as powerful proxies for understanding complex data distributions (like scanpaths) and quantifying the importance of different factors, which is difficult with simpler, mechanistic models.
- Saccade Statistics are Well-Reproduced: DeepGaze III accurately reproduces many statistical properties of human scanpaths, including saccade amplitude and direction distributions, and the angle between saccades.
- Spatiotopic vs. Retinotopic Priority: The model's performance suggests that for free viewing, a "spatiotopic" spatial priority map (independent of foveal location) is sufficient, rather than a "retinotopic" one that depends on the current gaze position.
Key Results
- State-of-the-Art Performance: DeepGaze III achieved new state-of-the-art results on the MIT1003, MIT300, and CAT2000 datasets, significantly outperforming previous models in metrics like log-likelihood, AUC, and NSS.
- On MIT1003, it achieved a log-likelihood of 2.442 bit/fixation, a substantial improvement over the next best model.
- On MIT300, it achieved 2.248 bit/fixation.
- On CAT2000, it reached 3.064 bit/fixation.
- Ablation Study Insights:
- Access to image content improved performance by 1.04 bit/fix over a model without it.
- Adding scanpath history (up to four previous fixations) further increased performance by 0.50 bit/fix.
- The influence of scanpath history decreased significantly after the second fixation.
- Center Bias Decomposition: The model could explain a significant portion of the "center bias" (tendency to look at the image center), with image content contributing about 30.8% and scanpath history about 22.6%.
Limitations and Future Work
- Scanpath Statistics Nuances: While DeepGaze III reproduces many scanpath statistics well, it doesn't perfectly capture all aspects, such as the exact tendency for vertical saccades or the peak at 0° for angles between saccades.
- Image-Specific Failures: The model still misses scanpath structure in some specific images, indicating room for improvement.
- Limited Scanpath History: The model currently uses only the last four fixations. Future work could explore incorporating more history or fixation durations.
- Task-Specific Scanpaths: The current model is for free viewing. Extending it to task-driven scenarios (like visual search) is a significant future direction, requiring new datasets and potentially more complex interactions between scene content and scanpath history.
- Interpretability: While modular, the deep learning architecture itself remains a "black box" to some extent. Future work could focus on using DeepGaze III to identify interesting cases for further qualitative analysis or hypothesis generation.
Summary updated