Beyond Universal Saliency: Personalized Saliency Prediction with Multi-task CNN

Yanyu Xu, Nianyi Li, Junru Wu, Jingyi Yu, Shenghua Gao

Abstract

Traditional saliency models aim for a "Universal Saliency Map" (USM), representing the average gaze of a population. However, individual gaze patterns vary significantly based on personal interests and attributes. This paper introduces the first personalized saliency dataset and a novel multi-task CNN architecture. By modeling personalized saliency as a residual of the universal map, the proposed framework efficiently predicts individual gaze patterns, outperforming standard universal models.

Introduction: The Case for Personalization

Saliency detection identifies image regions that naturally attract human attention. While universal models capture general trends, they fail to account for individual differences. For instance, in a complex scene, one observer might focus on text while another looks at faces or specific objects.

Individual gaze patterns vary significantly across different observers for the same image.

The primary obstacles to personalized saliency have been the lack of large-scale individual gaze data and the computational cost of training separate models for every user.

Methodology: Residual Learning and Multi-task CNNs

The authors propose that a Personalized Saliency Map (PSM) can be decomposed into a Universal Saliency Map (USM) and a person-specific discrepancy ($\Delta$):

$$PSM = USM + \Delta$$

This approach, rooted in residual learning, simplifies the task: the network learns to "correct" a baseline rather than predicting complex patterns from scratch.

Architecture

The proposed Multi-task CNN consists of:

  1. Shared Bottom Layers: These layers extract general visual features from a 4-channel input (RGB + USM).
  2. Multi-Branch Top Layers: The network branches out, with each branch dedicated to a specific individual to learn their unique $\Delta$.
  3. Intermediate Supervision: Loss layers at conv5 and conv6 provide "hints" during training, ensuring robust feature representation.
The Multi-task CNN architecture features shared bottom layers and individual branches for personalized saliency prediction.

The PSM Dataset

To support this research, the authors created the PSM Dataset, featuring 1,600 images viewed by 20 subjects. Crucially, each subject viewed every image four times over a week to ensure the ground truth reflects stable personal preferences rather than transient glances. The dataset includes 26,100 manually segmented objects across 242 classes, allowing for detailed semantic analysis of gaze discrepancies.

Key Findings

  • Semantic Drivers: While faces attract universal attention, objects like watches or specific text are the primary sources of individual gaze variation.
  • Efficiency of Residuals: Using the USM as an input significantly improves performance compared to RGB-only models, confirming that the USM provides a valuable starting point.
  • Supervision Benefits: Middle-layer supervision was found to measurably boost accuracy, helping the deep model converge on meaningful features.

Results and Future Directions

The Multi-task CNN achieved state-of-the-art results, with an AUC Judd of 85.94 and a Correlation Coefficient (CC) of 76.28.

Future work aims to address the "new user" problem—where a branch must be added for each new individual—by potentially predicting the discrepancy ($\Delta$) from observer metadata such as age, gender, or occupation, eliminating the need for prior gaze data collection.

Summary updated

Abstract

Saliency detection is a long standing problem in computer vision. Tremendous efforts have been focused on exploring a universal saliency model across users despite their differences in gender, race, age, etc. Yet recent psychology studies suggest that saliency is highly specific than universal: individuals exhibit heterogeneous gaze patterns when viewing an identical scene containing multiple salient objects. In this paper, we first show that such heterogeneity is common and critical for reliable saliency prediction. Our study also produces the first database of personalized saliency maps (PSMs). We model PSM based on universal saliency map (USM) shared by different participants and adopt a multi-task CNN framework to estimate the discrepancy between PSM and USM. Comprehensive experiments demonstrate that our new PSM model and prediction scheme are effective and reliable.
BibTeX
@inproceedings{xu2017beyond,
  author = {Yanyu Xu and Nianyi Li and Junru Wu and Jingyi Yu and Shenghua Gao},
  title = {{Beyond Universal Saliency: Personalized Saliency Prediction with Multi-task CNN}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/543},
}
Figure