Few-Shot Personalized Saliency Prediction Using Tensor Regression for Preserving Structural Global Information

Yuya Moroto, Keisuke Maeda, Takahiro Ogawa, Miki HaseyamaarXiv 2023 · arXiv · OpenReview

Abstract

This paper presents few-shot personalized saliency prediction based on inter-personnel gaze patterns. In contrast to a general saliency map, a personalized saliency map (PSM) has great potential since its map indicates the person-specific visual attention that is useful for obtaining individual visual preferences from heterogeneity of gazed areas. The PSM prediction is needed for acquiring the PSM for the unseen image, but its prediction is still a challenging task due to the complexity of individual gaze patterns. For modeling individual gaze patterns for various images, although the eye-tracking data obtained from each person is necessary to construct PSMs, it is difficult to acquire the massive amounts of such data. Here, one solution for efficient PSM prediction from the limited amount of data can be the effective use of eye-tracking data obtained from other persons. In this paper, to effectively treat the PSMs of other persons, we focus on the effective selection of images to acquire eye-tracking data and the preservation of structural information of PSMs of other persons. In the experimental results, we confirm that the above two focuses are effective for the PSM prediction with the limited amount of eye-tracking data.

Introduction

Standard saliency models produce a Universal Saliency Map (USM), which predicts what an "average" person finds interesting. However, human attention is highly individual; a professional scout, a casual hiker, and a botanist will look at the same forest scene very differently. To capture these nuances, researchers use Personalized Saliency Maps (PSMs).

The primary challenge is that state-of-the-art deep learning models typically require thousands of eye-tracking samples per person to learn these individual preferences. Collecting this much data is a massive burden for users. This paper tackles the "few-shot" problem: how to accurately predict a person's unique gaze patterns using only a small amount of their previous eye-tracking data (approximately 100 images).

Methodology

The proposed framework moves away from traditional vector-based regression to a more spatially-aware tensor approach through a three-phase pipeline:

  1. Multi-Task CNN for Training Persons: A model is trained on a large dataset of "training persons" to learn the general relationship between image features and diverse gaze tendencies.
  2. Adaptive Image Selection (AIS): To characterize a new "target" person, the system selects "common images" that elicit the most diverse gaze patterns among the training group. These high-variance images act as a diagnostic tool to quickly identify which training persons the new user most resembles.
  3. Tensor-to-Matrix Regression: This is the core innovation. Most regression models flatten 2D heatmaps into 1D vectors, which destroys the spatial relationships between pixels. The authors use tensor-based regression, treating the PSMs of training persons as a multi-dimensional array (a tensor). This allows the model to learn weights that preserve the two-dimensional spatial configuration both within a single map and between different people's maps.
The proposed framework for few-shot personalized saliency prediction, illustrating the Multi-task CNN, Adaptive Image Selection (AIS), and Tensor-to-matrix regression pipeline.

Key Innovation: Preserving Spatial Structure

A critical insight of this work is that saliency is a structured distribution rather than a collection of independent pixels. By using tensor regression instead of vector regression, the model respects the fact that if a person looks at a specific coordinate, they are likely to look at neighboring coordinates as well. The model learns an optimal weighted combination of many training persons, capturing interpersonal relationships implicitly.

Comparison between traditional vector-input regression, which uses raster scanning and loses spatial context, and the proposed tensor-input regression that preserves relationships within and between PSMs.

Experimental Results

The study utilized a dataset containing 1,600 images viewed by 30 participants. The proposed method was evaluated using KL Divergence (distributional accuracy) and Correlation Coefficient (CC) (pixel-wise similarity).

  • Distribution Accuracy: The proposed method achieved a KL divergence score of 1.00, significantly outperforming universal models (~3.57) and standard few-shot baselines (1.82 to 7.64).
  • Pixel Similarity: In terms of CC, the method scored 0.775. While slightly lower than the "Object-Gaze Similarity" (OGS) baseline (0.781), the method is more versatile because it does not require explicit object labels and performs better on the overall distribution (KL divergence).
  • Hyperparameter Impact: Results showed that increasing the "rank" ($R$) of the tensor regression improves performance up to a point ($R=50$), after which it plateaus.

Conclusion

This research demonstrates that tensor-based regression combined with adaptive image selection effectively enables few-shot personalized saliency prediction. By preserving the structural global information of gaze patterns, the model achieves high distributional accuracy without needing semantic object labels. Future work will explore alternative tensor rank decompositions to better balance detail and computational speed.

Summary updated

Abstract

This paper presents few-shot personalized saliency prediction based on inter-personnel gaze patterns. In contrast to a general saliency map, a personalized saliecny map (PSM) has been great potential since its map indicates the person-specific visual attention that is useful for obtaining individual visual preferences from heterogeneity of gazed areas. The PSM prediction is needed for acquiring the PSM for the unseen image, but its prediction is still a challenging task due to the complexity of individual gaze patterns. For modeling individual gaze patterns for various images, although the eye-tracking data obtained from each person is necessary to construct PSMs, it is difficult to acquire the massive amounts of such data. Here, one solution for efficient PSM prediction from the limited amount of data can be the effective use of eye-tracking data obtained from other persons. In this paper, to effectively treat the PSMs of other persons, we focus on the effective selection of images to acquire eye-tracking data and the preservation of structural information of PSMs of other persons. In the experimental results, we confirm that the above two focuses are effective for the PSM prediction with the limited amount of eye-tracking data.
BibTeX
@misc{moroto2023few,
  author = {Yuya Moroto and Keisuke Maeda and Takahiro Ogawa and Miki Haseyama},
  title = {{Few-Shot Personalized Saliency Prediction Using Tensor Regression for Preserving Structural Global Information}},
  year = {2023},
  eprint = {2307.02799},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.02799},
}
Figure