Abstract
Traditional saliency models aim for a "Universal Saliency Map" (USM), representing the average gaze of a population. However, individual gaze patterns vary significantly based on personal interests and attributes. This paper introduces the first personalized saliency dataset and a novel multi-task CNN architecture. By modeling personalized saliency as a residual of the universal map, the proposed framework efficiently predicts individual gaze patterns, outperforming standard universal models.
Introduction: The Case for Personalization
Saliency detection identifies image regions that naturally attract human attention. While universal models capture general trends, they fail to account for individual differences. For instance, in a complex scene, one observer might focus on text while another looks at faces or specific objects.
The primary obstacles to personalized saliency have been the lack of large-scale individual gaze data and the computational cost of training separate models for every user.
Methodology: Residual Learning and Multi-task CNNs
The authors propose that a Personalized Saliency Map (PSM) can be decomposed into a Universal Saliency Map (USM) and a person-specific discrepancy ($\Delta$):
$$PSM = USM + \Delta$$
This approach, rooted in residual learning, simplifies the task: the network learns to "correct" a baseline rather than predicting complex patterns from scratch.
Architecture
The proposed Multi-task CNN consists of:
- Shared Bottom Layers: These layers extract general visual features from a 4-channel input (RGB + USM).
- Multi-Branch Top Layers: The network branches out, with each branch dedicated to a specific individual to learn their unique $\Delta$.
- Intermediate Supervision: Loss layers at
conv5andconv6provide "hints" during training, ensuring robust feature representation.
The PSM Dataset
To support this research, the authors created the PSM Dataset, featuring 1,600 images viewed by 20 subjects. Crucially, each subject viewed every image four times over a week to ensure the ground truth reflects stable personal preferences rather than transient glances. The dataset includes 26,100 manually segmented objects across 242 classes, allowing for detailed semantic analysis of gaze discrepancies.
Key Findings
- Semantic Drivers: While faces attract universal attention, objects like watches or specific text are the primary sources of individual gaze variation.
- Efficiency of Residuals: Using the USM as an input significantly improves performance compared to RGB-only models, confirming that the USM provides a valuable starting point.
- Supervision Benefits: Middle-layer supervision was found to measurably boost accuracy, helping the deep model converge on meaningful features.
Results and Future Directions
The Multi-task CNN achieved state-of-the-art results, with an AUC Judd of 85.94 and a Correlation Coefficient (CC) of 76.28.
Future work aims to address the "new user" problem—where a branch must be added for each new individual—by potentially predicting the discrepancy ($\Delta$) from observer metadata such as age, gender, or occupation, eliminating the need for prior gaze data collection.
Summary updated