High-Level Concepts for Affective Understanding of Images

Afsheen Rafaqat Ali, Usman Shahid, Mohsen Ali, Jeffrey Ho

TL;DR

This paper bridges the gap between image content and human emotional response by explicitly using High-Level Concepts (HLCs) like objects and scenes, moving beyond low-level features or black-box CNNs. They propose a novel linear admixture model to understand HLC-emotion associations and a hybrid ensemble model for accurate emotion prediction, achieving competitive results with state-of-the-art methods.

What Problem Does This Paper Address?

The paper tackles the challenge of enabling machines to understand and predict the emotional response that images evoke in humans. While current AI can analyze image content (e.g., identify objects), it often lacks the "emotional intelligence" to grasp how that content makes a viewer feel. Previous approaches either relied solely on low-level visual features (like color and texture), which are insufficient for capturing nuanced emotions, or treated powerful Convolutional Neural Networks (CNNs) as black boxes, offering little insight into why they make certain predictions. This "affective gap" limits applications like more empathetic image retrieval or captioning.

Methods

The core of this work lies in leveraging High-Level Concepts (HLCs) – semantically meaningful information like objects (e.g., "butterfly," "toilet seat") and scenes (e.g., "garden," "garbage dump") – extracted from images.

  1. HLC Extraction: Pre-trained CNNs, specifically those trained on ImageNet (for object recognition) and Places205 (for scene recognition), are used to generate HLC probability vectors for each image.
  2. Linear Admixture Model: Inspired by topic modeling in text analysis, the authors propose a linear model where an image's HLC distribution is seen as a mixture of HLC distributions associated with different emotion classes (Anger, Disgust, Fear, Joy, Sadness, Surprise, Neutral). This model helps uncover explicit associations between specific HLCs and emotions.
  3. Hybrid Model (Ensemble of Regressors): Recognizing that different emotions might be better represented by different HLCs, they develop a "Hybrid Model." This involves training separate Support Vector Regression (SVR) models for each emotion class, with each SVR using a different subset of HLCs and low-level features. An ensemble of these specialized regressors then predicts the final emotion distribution.

Datasets

The paper uses two established datasets for affective computing:

  • ArtPhoto [12]: Contains 807 artistic photographs across eight emotion classes, where images were intentionally created to evoke specific emotions.
  • Emotion6 [15]: A larger dataset with 1980 images, each associated with a seven-dimensional emotion probability distribution (six basic Ekman emotions plus neutral), reflecting a more realistic scenario where images can evoke mixed emotions.

Key Insights and Ideas

  • HLCs are Crucial for Affective Understanding: The research strongly suggests that high-level semantic concepts (objects, scenes) are essential for bridging the gap between image content and emotional response, offering more interpretability than low-level features alone.
  • Emotion-Specific Feature Relevance: Different emotions are not equally associated with all HLCs. For instance, "garbage dump" might strongly correlate with "Disgust," while "flowers" and "gardens" are linked to "Joy." This insight motivates the use of tailored feature subsets for each emotion.
  • Interpretability is Key: By explicitly modeling HLC-emotion associations, the approach provides a clearer understanding of why an image might elicit a certain emotion, a significant improvement over black-box CNN methods.
  • Hybrid Approach Yields Best Results: Combining specialized regressors (each trained on the most relevant features for a specific emotion) into an ensemble model significantly improves prediction accuracy and quality.

Key Results

  • Competitive Performance: The proposed Hybrid Model achieves classification accuracy of 52% on the Emotion6 dataset, outperforming previous state-of-the-art results reported for this dataset [15].
  • Improved Distribution Prediction: The Hybrid Model also shows improved quality in predicting the full emotion distribution vector compared to methods using a single feature set.
  • Valence-Arousal Prediction: The Hybrid Model also demonstrates strong performance in predicting Valence-Arousal (VA) scores, outperforming other regressors and even previous methods that mapped low-level features to VA.
  • Visualizing Associations: The paper presents heatmaps (Figure 2) that visually demonstrate the learned associations between specific HLCs and emotion classes, offering valuable insights into the underlying relationships.

Limitations and Future Work

  • HLC Scope: The current work primarily uses object and place concepts. The authors acknowledge that a broader set of HLCs, including human actions, poses, and facial expressions, could further enhance performance.
  • Dependency on Pre-trained Classifiers: The HLC extraction relies on pre-trained CNNs. The accuracy of HLC detection is thus dependent on the performance of these existing classifiers, which can sometimes produce incorrect classifications.
  • Future Directions: The authors plan to expand the scope of HLCs to include more diverse semantic information. They also aim to explore more sophisticated models that can better capture the complex relationships between HLCs and emotions.

Summary updated

Abstract

This paper aims to bridge the affective gap between image content and the emotional response of the viewer it elicits by using High-Level Concepts (HLCs). In contrast to previous work that relied solely on low-level features or used convolutional neural network (CNN) as a black-box, we use HLCs generated by pretrained CNNs in an explicit way to investigate the relations/associations between these HLCs and a (small) set of Ekman's emotional classes. As a proof-of-concept, we first propose a linear admixture model for modeling these relations, and the resulting computational framework allows us to determine the associations between each emotion class and certain HLCs (objects and places). This linear model is further extended to a nonlinear model using support vector regression (SVR) that aims to predict the viewer's emotional response using both low-level image features and HLCs extracted from images. These class-specific regressors are then assembled into a regressor ensemble that provide a flexible and effective predictor for predicting viewer's emotional responses from images. Experimental results have demonstrated that our results are comparable to existing methods, with a clear view of the association between HLCs and emotional classes that is ostensibly missing in most existing work.
BibTeX
@inproceedings{ali2017high,
  author = {Afsheen Rafaqat Ali and Usman Shahid and Mohsen Ali and Jeffrey Ho},
  title = {{High-Level Concepts for Affective Understanding of Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2017},
  url = {https://openreview.net/forum?id=Zb44py4fOJJn},
}
Figure