TL;DR
This paper explores how to automatically recognize the "pleasantness" of natural images, a key aspect of affective computing. By combining affordable eye-tracking data with deep learning visual features in a realistic, less controlled setting, they found that while eye movements alone struggle, visual category information is a surprisingly strong predictor of pleasantness, hinting at potential biases in current deep learning models.
What Problem Does This Paper Address?
Understanding and recognizing emotions is a long-standing goal in AI, crucial for more natural human-computer interaction (HCI). While recognizing a user's emotional state is one aspect, recognizing the emotional "gist" or pleasantness of an image itself is another, important for applications like content analysis and personalized recommendations. Traditional computer vision methods have struggled with natural image pleasantness compared to psychophysical signals like eye movements. However, eye-tracking technology has historically been expensive, limiting its widespread use. The authors aim to bridge this gap by investigating the effectiveness of affordable eye-tracking combined with modern deep learning techniques for recognizing image pleasantness, specifically in a "loosely controlled" HCI scenario that mimics everyday use.
Methods
The researchers employed a multi-faceted approach:
- New Eye Movement Dataset: They collected a novel dataset of eye movements from 20 participants viewing 382 affective images using an affordable Tobii EyeX eye tracker. This setup aimed to mimic real-world HCI by allowing participants to self-calibrate and avoid strict environmental controls.
- Deep Convolutional Neural Networks (CNNs): They leveraged pre-trained deep CNNs (VGG16 and VGG19) to extract visual features from the images. This is a standard approach for capturing rich visual information, moving beyond handcrafted features.
- Eye Movement Feature Extraction: They extracted various features from the eye-tracking data, including fixation duration, saccade length and slope, and fixation density maps. They also explored inter-observer visual congruency as a measure of agreement.
- Classification: Linear Support Vector Machines (SVMs) were used as classifiers. They employed a one-vs-rest strategy for the three-class problem (unpleasant, neutral, pleasant) and used repeated 10-fold cross-validation with a resampling strategy to handle imbalanced data.
- Feature Fusion: They investigated combining eye movement features with visual features to see if a synergistic effect improved performance.
Datasets
- International Affective Picture System (IAPS) Corpus: The study utilized images from the IAPS corpus, a widely recognized dataset for affective image research.
- A subset of 95 images with high emotional agreement between genders and a single visual class category ("People & Daily Activity") was used, consistent with prior work [6].
- A larger set of 296 images with gender agreement on emotional gist but diverse visual categories was also used.
- The full set of 382 images, including those with gender disagreement and a wide range of visual categories, was also analyzed.
- New Eye Movement Data: The paper introduces a new dataset of eye movements collected from 20 participants viewing these IAPS images using an affordable Tobii EyeX eye tracker. This dataset is made available for research.
Key Insights and Ideas
- Loosely Controlled HCI is Challenging: Recognizing image pleasantness from eye movements becomes significantly harder in a less controlled, everyday HCI setting. Previously successful techniques struggled, highlighting the impact of environmental factors and user interaction styles.
- Visual Category is a Strong Predictor: Surprisingly, the visual category of an image (e.g., "dog," "car," "food") proved to be a powerful cue for predicting pleasantness, even outperforming some eye movement features. This suggests that the semantic content of an image is deeply intertwined with its emotional valence.
- Potential Bias in Deep Learning: The strong performance of visual categories raises a flag. It suggests that current deep learning models, trained on large datasets, might be implicitly learning these category-based biases rather than a more nuanced understanding of visual aesthetics or emotional response. This could lead to models that perform well on common categories but fail on more diverse or unusual ones.
- Affordable Eye-Tracking Limitations: While affordable eye trackers increase accessibility, their performance characteristics (e.g., sampling rate) can introduce noise, making it more difficult to extract reliable features compared to high-end systems.
Key Results
- Eye Movements Alone: In the loosely controlled setup, eye movement features alone did not perform significantly above chance (33.3%) for pleasantness recognition across most scenarios. The fixation density map showed promise on a subset of images (296 images) with high gender agreement and diverse visual categories, achieving 37.8% accuracy.
- Deep Visual Features: Deep CNN features (VGG16/VGG19) significantly outperformed traditional visual features (PHOW) and eye movement features. For the 296 and 382 image sets, deep features achieved accuracies around 42-44%, significantly above chance.
- Visual Category Detection: Using detected visual categories as features, performance was strong, especially with diverse visual categories. For the 382 images, this approach achieved accuracies around 42-43%.
- Feature Fusion: Combining eye movement features with deep visual features did not yield significant improvements over using deep features alone, suggesting that the eye movement signals in this less controlled setting were not adding substantial information.
Limitations and Future Work
- Noise in Affordable Eye-Tracking: The use of an affordable eye tracker with a lower sampling rate introduced noise, making feature extraction more challenging. The authors note that scaling up inference from eye movements with current affordable technologies might be difficult.
- Impact of Less Restrictive Setup: The less controlled experimental setup, mimicking everyday HCI, made the task significantly harder than in previous, more controlled studies. This highlights the importance of experimental design in affective computing.
- Gender Agreement and Visual Categories: The performance of features varied depending on the level of gender agreement on emotional gist and the diversity of visual categories in the image sets.
- Future Work: The authors suggest further investigation into the role of visual categories and their potential bias in deep learning models. They also plan to address this bias in future work. Additionally, they acknowledge the need for more robust machine learning techniques and reliable sensors for eye movement analysis in HCI.
Summary updated