Tensor-Based Emotional Category Classification via Visual Attention-Based Heterogeneous CNN Feature Fusion

Yuya Moroto, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama

TL;DR

This paper introduces a novel method for classifying emotions evoked by images by fusing visual features from multiple CNNs with eye-gaze data. By representing visual attention over time as a tensor, the method captures dynamic emotional responses and achieves improved classification accuracy compared to existing approaches.

What Problem Does This Paper Address?

Understanding human emotions evoked by images is a complex task. While image-based features (like textures) are well-handled by deep learning, human-based information, such as where a person looks and how their attention shifts over time, is crucial for capturing emotional nuances. Existing methods often struggle to effectively integrate these dynamic, human-centric aspects with powerful image features. Specifically, the paper addresses the challenge of:

  • Integrating visual and attentional information: Emotions are linked to both what's in an image and how a person visually engages with it.
  • Capturing temporal dynamics: Human emotional responses aren't static; they evolve as we look at an image.
  • Leveraging diverse visual features: Different CNN architectures extract complementary information, but fusing these heterogeneous features effectively is difficult.
  • Limited labeled data: Training deep models often requires vast amounts of data, which can be a bottleneck for specialized tasks like emotion classification with gaze data.

Methods

The core of the proposed method lies in its novel approach to feature fusion and representation:

  • Gaze-Based Image Representation: The paper creates a new image representation that combines the original image with eye-gaze data. This involves generating fixation maps from gaze locations and durations, then using these to create "gaze and image weight" (GIW) matrices. These GIW matrices are then used to create a fourth-order tensor that captures both spatial image information and temporal gaze patterns.
  • Heterogeneous CNN Feature Fusion (CFT): Instead of using a single CNN, the method extracts features from multiple pre-trained CNNs (DenseNet201, InceptionResNet-v2, Xception). Since these features have different dimensions, they are first reduced to a common dimension using Fisher Discriminant Analysis (FDA). These aligned features are then combined into a "CNN Feature-based Tensor" (CFT). This tensor structure allows for the simultaneous consideration of different CNN features, their interactions, and the temporal gaze information.
  • Tensor Analysis for Classification:
    • General Tensor Discriminant Analysis (GTDA): This technique is applied to the CFT to transform the features into a more discriminative space, enhancing their suitability for classification. GTDA helps to maximize class separability within the tensor representation.
  • Logistic Tensor Regression (LTR): Finally, the transformed features are used with LTR to perform the actual emotional category classification. LTR is a regression model adapted for tensor inputs, allowing for the estimation of emotional labels.

Datasets

  • Abstract Painting Dataset: The experiment utilized a dataset of 280 abstract paintings, annotated with eight emotional categories (awe, amusement, contentment, anger, excitement, sad, disgust, and fear). This dataset was previously collected and annotated by Machajdik et al. [29].
  • Eye Gaze Data: Eye gaze data was collected from 13 human subjects (11 males, 2 females, aged 22-26) who viewed the abstract paintings.

Key Insights and Ideas

  • Visual Attention as a Temporal Signal for Emotion: The paper strongly suggests that how a person looks at an image over time is as informative as what they look at, and this dynamic aspect is crucial for understanding evoked emotions.
  • Tensor Representation for Multi-Modal Fusion: Representing fused features as a tensor is a powerful way to capture complex interactions between different modalities (CNN features) and temporal dynamics (gaze). This goes beyond simple concatenation.
  • Heterogeneous CNNs Offer Complementary Information: Using multiple CNNs, even with dimensionality reduction, provides richer feature sets than relying on a single architecture. The tensor fusion allows these diverse features to work together.
  • Gaze Data Bridges the Gap for Limited Datasets: The method's reliance on gaze data, combined with pre-trained CNNs, makes it more robust to smaller, specialized datasets, which is a common challenge in emotion recognition research.

Key Results

The proposed method (PM) demonstrated superior performance compared to several comparative methods.

  • Overall Improvement: The PM achieved an average F1-measure of approximately 0.59, which is about 10% improvement over state-of-the-art comparative methods.
  • Effectiveness of Gaze and Temporal Information: The PM significantly outperformed methods that did not consider temporal changes in visual attention (e.g., CM1, CM2, CM3, CM4, CM5).
  • Benefit of Heterogeneous Fusion: The PM also outperformed methods that used simpler concatenation or vector-based fusion of CNN features (CM6, CM7, CM8).
  • Statistical Significance: Welch's t-test confirmed that the PM's performance was statistically superior to most comparative methods (p < 0.05 for CMs 1-5 and 7).

Limitations and Future Work

  • Dataset Size: The study used only one abstract painting dataset due to the burden of collecting eye gaze data. The authors suggest using other datasets in the future to verify the robustness of their method.
  • Optimal CNN Selection: The paper used three specific CNNs, and the authors note that determining the optimal kinds and number of CNN features for fusion is an area for future investigation.
  • Order of CNN Features: The order in which CNN features are combined in the tensor can influence performance, and the paper suggests exploring methods for deciding this order.
  • Real-world Applicability: While abstract paintings are good for studying emotion, future work could explore the method's effectiveness on more diverse image types.

Summary updated

Abstract

The paper proposes a method of visual attention-based emotion classification through eye gaze analysis. Concretely, tensor-based emotional category classification via visual attention-based heterogeneous convolutional neural network (CNN) feature fusion is proposed. Based on the relationship between human emotions and changes in visual attention with time, the proposed method performs new gaze-based image representation that is suitable for reflecting the characteristics of the changes in visual attention with time. Furthermore, since emotions evoked in humans are closely related to objects in images, our method uses a CNN model to obtain CNN features that can represent their characteristics. For improving the representation ability to the emotional categories, we extract multiple CNN features from our novel gaze-based image representation and enable their fusion by constructing a novel tensor consisting of these CNN features. Thus, this tensor construction realizes the visual attention-based heterogeneous CNN feature fusion. This is the main contribution of this paper. Finally, by applying logistic tensor regression with general tensor discriminant analysis to the newly constructed tensor, the emotional category classification becomes feasible. Since experimental results show that the proposed method enables the emotional category classification with the F1-measure of approximately 0.6, and about 10% improvement can be realized compared to comparative methods including state-of-the-art methods, the effectiveness of the proposed method is verified.
BibTeX
@inproceedings{moroto2020tensor,
  author = {Yuya Moroto and Keisuke Maeda and Takahiro Ogawa and Miki Haseyama},
  title = {{Tensor-Based Emotional Category Classification via Visual Attention-Based Heterogeneous CNN Feature Fusion}},
  booktitle = {Sensors},
  year = {2020},
  url = {https://openreview.net/forum?id=r3UCYjCJO1},
}
Figure