Few-Shot Personalized Saliency Prediction Based on Adaptive Image Selection Considering Object and Visual Attention

Yuya Moroto, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama

TL;DR

This paper tackles the challenge of predicting personalized visual attention (saliency maps) with very limited data from a new user. They introduce an "Adaptive Image Selection" (AIS) method that intelligently picks training images to maximize diversity and variance in attention patterns, enabling accurate predictions with fewer samples.

What Problem Does This Paper Address?

Predicting where a person will look on an image (saliency prediction) is useful for many image processing tasks. While general saliency maps (Universal Saliency Maps or USMs) capture common attention patterns, individual preferences vary significantly. Personalized Saliency Maps (PSMs) aim to capture these individual differences. However, existing methods for predicting PSMs require a large amount of gaze data from the target individual, which is burdensome and often impractical. This paper addresses the "few-shot" problem: how to predict a PSM for a new person with only a small amount of their gaze data. The core challenge is efficiently learning these individual preferences without overwhelming the user with data collection.

Methods

The proposed method, Few-Shot Personalized Saliency Prediction (FPSP), builds upon a pre-trained multi-task Convolutional Neural Network (CNN) capable of predicting PSMs for multiple individuals. The novelty lies in their Adaptive Image Selection (AIS) scheme. Instead of randomly selecting images for the target person to view, AIS strategically chooses images from a dataset based on two criteria:

  1. Object Diversity: AIS uses YOLO-v3 object detection to identify various objects within images. It prioritizes images containing a diverse range of objects to ensure the model learns about different visual elements that might attract attention.
  2. Saliency Variance: AIS analyzes the variance of PSMs for objects across different individuals in the dataset. Images with objects that elicit highly varied attention patterns (some people look, others don't) are favored. This helps in distinguishing individual preferences from common attention.

By selecting images that are rich in diverse objects and exhibit high saliency variance, AIS aims to maximize the information gained from each viewed image, thereby enabling accurate PSM prediction with fewer samples. The person similarity is then calculated based on the gaze data from these selected images, and this similarity is used to combine predictions from similar individuals to form the target person's PSM.

Datasets

  • PSM Dataset: This dataset, consisting of 1600 images and corresponding gaze data from 30 individuals, is used for training the multi-task CNN and for selecting images via the AIS method.
  • MIT Saliency Benchmark: Used for comparison with USM prediction methods.
  • SALICON dataset: Used for training the SalGAN method.

Key Insights and Ideas

  • The burden of data collection is a major bottleneck for personalized models: Requiring extensive gaze data from every new user is impractical.
  • Intelligent image selection is crucial for few-shot learning: Not all training data is equally informative. Selecting data that maximizes diversity and variance is key to efficient learning.
  • Object diversity and saliency variance are complementary signals: Combining information about what is in an image (objects) with how people attend to it (saliency variance) provides a powerful way to differentiate individual preferences.
  • YOLO-v3 is effective for object detection in this context: Its ability to identify a wide range of objects aids in ensuring visual diversity in selected training images.

Key Results

  • The proposed FPSP method with AIS significantly outperforms existing USM prediction methods and baseline PSM prediction methods in terms of Pearson's correlation coefficient (CC), similarity (Sim), and Kullback-Leibler divergence (KLdiv).
  • In quantitative evaluations, FPSP achieved an average CC of 0.7845, Sim of 0.6557, and KLdiv of 1.083, outperforming all comparative methods.
  • Qualitative results show that FPSP produces PSMs that are most similar to the ground truth for individual users.
  • The AIS method is shown to be more effective than other image selection strategies (ISVF, ISPSM) for PSM prediction.
  • The robustness of FPSP based on AIS is demonstrated, showing consistent performance even with a small number of training images (as few as 10).

Limitations and Future Work

  • Determination of threshold τ: The paper mentions that the determination of the threshold value τ for person similarity will be investigated in future work.
  • Computational cost of object detection: While YOLO-v3 is effective, its computational cost might be a consideration for real-time applications.
  • Generalizability to diverse user groups: The study uses a specific PSM dataset; further validation on more diverse populations might be beneficial.

The paper concludes that FPSP based on AIS effectively predicts personalized saliency maps with a small number of training images, significantly reducing the burden on users for data collection.

Summary updated

Abstract

A few-shot personalized saliency prediction based on adaptive image selection considering object and visual attention is presented in this paper. Since general methods predicting personalized saliency maps (PSMs) need a large number of training images, the establishment of a theory using a small number of training images is needed. To tackle this problem, although finding persons who have visual attention similar to that of a target person is effective, all persons have to commonly gaze at many images. Thus, it becomes difficult and unrealistic when considering their burden. On the other hand, this paper introduces a novel adaptive image selection (AIS) scheme that focuses on the relationship between human visual attention and objects in images. AIS focuses on both a diversity of objects in images and a variance of PSMs for the objects. Specifically, AIS selects images so that selected images have various kinds of objects to maintain their diversity. Moreover, AIS guarantees the high variance of PSMs for persons since it represents the regions that many persons commonly gaze at or do not gaze at. The proposed method enables selecting similar users from a small number of images by selecting images that have high diversities and variances. This is the technical contribution of this paper. Experimental results show the effectiveness of our personalized saliency prediction including the new image selection scheme.
BibTeX
@inproceedings{moroto2020few,
  author = {Yuya Moroto and Keisuke Maeda and Takahiro Ogawa and Miki Haseyama},
  title = {{Few-Shot Personalized Saliency Prediction Based on Adaptive Image Selection Considering Object and Visual Attention}},
  booktitle = {Sensors},
  year = {2020},
  url = {https://openreview.net/forum?id=Tvkr22lM085},
}
Figure