Investigating Natural Image Pleasantness Recognition using Deep Features and Eye Tracking for Loosely Controlled Human-computer Interaction

Hamed Rezazadegan Tavakoli, Jorma Laaksonen, Esa RahtuarXiv 2017 · OpenReview

TL;DR

This paper explores how to automatically recognize the "pleasantness" of natural images, a key aspect of affective computing. By combining affordable eye-tracking data with deep learning visual features in a realistic, less controlled setting, they found that while eye movements alone struggle, visual category information is a surprisingly strong predictor of pleasantness, hinting at potential biases in current deep learning models.

What Problem Does This Paper Address?

Understanding and recognizing emotions is a long-standing goal in AI, crucial for more natural human-computer interaction (HCI). While recognizing a user's emotional state is one aspect, recognizing the emotional "gist" or pleasantness of an image itself is another, important for applications like content analysis and personalized recommendations. Traditional computer vision methods have struggled with natural image pleasantness compared to psychophysical signals like eye movements. However, eye-tracking technology has historically been expensive, limiting its widespread use. The authors aim to bridge this gap by investigating the effectiveness of affordable eye-tracking combined with modern deep learning techniques for recognizing image pleasantness, specifically in a "loosely controlled" HCI scenario that mimics everyday use.

Methods

The researchers employed a multi-faceted approach:

  • New Eye Movement Dataset: They collected a novel dataset of eye movements from 20 participants viewing 382 affective images using an affordable Tobii EyeX eye tracker. This setup aimed to mimic real-world HCI by allowing participants to self-calibrate and avoid strict environmental controls.
  • Deep Convolutional Neural Networks (CNNs): They leveraged pre-trained deep CNNs (VGG16 and VGG19) to extract visual features from the images. This is a standard approach for capturing rich visual information, moving beyond handcrafted features.
  • Eye Movement Feature Extraction: They extracted various features from the eye-tracking data, including fixation duration, saccade length and slope, and fixation density maps. They also explored inter-observer visual congruency as a measure of agreement.
  • Classification: Linear Support Vector Machines (SVMs) were used as classifiers. They employed a one-vs-rest strategy for the three-class problem (unpleasant, neutral, pleasant) and used repeated 10-fold cross-validation with a resampling strategy to handle imbalanced data.
  • Feature Fusion: They investigated combining eye movement features with visual features to see if a synergistic effect improved performance.

Datasets

  • International Affective Picture System (IAPS) Corpus: The study utilized images from the IAPS corpus, a widely recognized dataset for affective image research.
  • A subset of 95 images with high emotional agreement between genders and a single visual class category ("People & Daily Activity") was used, consistent with prior work [6].
  • A larger set of 296 images with gender agreement on emotional gist but diverse visual categories was also used.
  • The full set of 382 images, including those with gender disagreement and a wide range of visual categories, was also analyzed.
  • New Eye Movement Data: The paper introduces a new dataset of eye movements collected from 20 participants viewing these IAPS images using an affordable Tobii EyeX eye tracker. This dataset is made available for research.

Key Insights and Ideas

  • Loosely Controlled HCI is Challenging: Recognizing image pleasantness from eye movements becomes significantly harder in a less controlled, everyday HCI setting. Previously successful techniques struggled, highlighting the impact of environmental factors and user interaction styles.
  • Visual Category is a Strong Predictor: Surprisingly, the visual category of an image (e.g., "dog," "car," "food") proved to be a powerful cue for predicting pleasantness, even outperforming some eye movement features. This suggests that the semantic content of an image is deeply intertwined with its emotional valence.
  • Potential Bias in Deep Learning: The strong performance of visual categories raises a flag. It suggests that current deep learning models, trained on large datasets, might be implicitly learning these category-based biases rather than a more nuanced understanding of visual aesthetics or emotional response. This could lead to models that perform well on common categories but fail on more diverse or unusual ones.
  • Affordable Eye-Tracking Limitations: While affordable eye trackers increase accessibility, their performance characteristics (e.g., sampling rate) can introduce noise, making it more difficult to extract reliable features compared to high-end systems.

Key Results

  • Eye Movements Alone: In the loosely controlled setup, eye movement features alone did not perform significantly above chance (33.3%) for pleasantness recognition across most scenarios. The fixation density map showed promise on a subset of images (296 images) with high gender agreement and diverse visual categories, achieving 37.8% accuracy.
  • Deep Visual Features: Deep CNN features (VGG16/VGG19) significantly outperformed traditional visual features (PHOW) and eye movement features. For the 296 and 382 image sets, deep features achieved accuracies around 42-44%, significantly above chance.
  • Visual Category Detection: Using detected visual categories as features, performance was strong, especially with diverse visual categories. For the 382 images, this approach achieved accuracies around 42-43%.
  • Feature Fusion: Combining eye movement features with deep visual features did not yield significant improvements over using deep features alone, suggesting that the eye movement signals in this less controlled setting were not adding substantial information.

Limitations and Future Work

  • Noise in Affordable Eye-Tracking: The use of an affordable eye tracker with a lower sampling rate introduced noise, making feature extraction more challenging. The authors note that scaling up inference from eye movements with current affordable technologies might be difficult.
  • Impact of Less Restrictive Setup: The less controlled experimental setup, mimicking everyday HCI, made the task significantly harder than in previous, more controlled studies. This highlights the importance of experimental design in affective computing.
  • Gender Agreement and Visual Categories: The performance of features varied depending on the level of gender agreement on emotional gist and the diversity of visual categories in the image sets.
  • Future Work: The authors suggest further investigation into the role of visual categories and their potential bias in deep learning models. They also plan to address this bias in future work. Additionally, they acknowledge the need for more robust machine learning techniques and reliable sensors for eye movement analysis in HCI.

Summary updated

Abstract

This paper revisits recognition of natural image pleasantness by employing deep convolutional neural networks and affordable eye trackers. There exist several approaches to recognize image pleasantness: (1) computer vision, and (2) psychophysical signals. For natural images, computer vision approaches have not been as successful as for abstract paintings and is lagging behind the psychophysical signals like eye movements. Despite better results, the scalability of eye movements is adversely affected by the sensor cost. While the introduction of affordable sensors have helped the scalability issue by making the sensors more accessible, the application of such sensors in a loosely controlled human-computer interaction setup is not yet studied for affective image tagging. On the other hand, deep convolutional neural networks have boosted the performance of vision-based techniques significantly in recent years. To investigate the current status in regard to affective image tagging, we (1) introduce a new eye movement dataset using an affordable eye tracker, (2) study the use of deep neural networks for pleasantness recognition, (3) investigate the gap between deep features and eye movements. To meet these ends, we record eye movements in a less controlled setup, akin to daily human-computer interaction. We assess features from eye movements, visual features, and their combination. Our results show that (1) recognizing natural image pleasantness from eye movement under less restricted setup is difficult and previously used techniques are prone to fail, and (2) visual class categories are strong cues for predicting pleasantness, due to their correlation with emotions, necessitating careful study of this phenomenon. This latter finding is alerting as some deep learning approaches may fit to the class category bias.
BibTeX
@misc{tavakoli2017investigating,
  author = {Hamed Rezazadegan Tavakoli and Jorma Laaksonen and Esa Rahtu},
  title = {{Investigating Natural Image Pleasantness Recognition using Deep Features and Eye Tracking for Loosely Controlled Human-computer Interaction}},
  year = {2017},
  eprint = {1704.02218},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1704.02218},
}
Figure