Smile upon the Face but Sadness in the Eyes: Emotion Recognition based on Facial Expressions and Eye Behaviors

Zhe ChenarXiv 2024 · DOI · arXiv

TL;DR

This paper introduces the EMER dataset, a novel multimodal dataset for emotion recognition that includes facial expressions and eye behaviors, aiming to bridge the "emotion gap" where facial expressions don't always reflect true emotions. They also propose EMERT, a Transformer-based architecture that effectively leverages both modalities to achieve state-of-the-art emotion recognition performance.

What Problem Does This Paper Address?

Emotion Recognition (ER) is crucial for human-computer interaction, but current methods heavily rely on Facial Expression Recognition (FER). The core problem is that facial expressions can be misleading or intentionally masked, leading to an "emotion gap" between what a person's face shows and their genuine emotional state. This gap can result in inaccurate ER. Existing multimodal ER datasets often use intrusive physiological sensors like EEG, which are costly and can affect participant behavior. This paper aims to address this by exploring the utility of non-invasive eye behaviors (eye movements and fixations) as a complementary signal to facial expressions for more robust ER.

Methods

The authors propose a novel Eye-behavior-based Multimodal Emotion Recognition Transformer (EMERT) architecture. EMERT has three main components:

  • Multimodal Feature Extraction (MFE): Extracts features from facial expressions (using ResNet), eye movements (using LSTM), and eye fixation maps (using LSTM).
  • Modality Adversarial Feature Decoupling (MAFD): Employs adversarial learning to separate modality-invariant emotion features from modality-specific ones. This helps to isolate features that are less susceptible to the "emotion gap" and those that capture unique emotional cues from each modality.
  • Emotion-aware Multi-task Transformer (EMT): A Transformer encoder that takes the decoupled features and learns modality-sensitive high-level representations. This component facilitates the complementary use of modality-invariant and modality-specific features, ultimately bridging the emotion gap. The EMT then uses separate prediction heads for ER and FER.

For FER annotation, they use an Active Learning-based Annotation (ALA) method that combines machine auto-annotation with manual expert annotation to ensure high reliability and efficiency.

Datasets

The paper introduces a new benchmark dataset:

  • EMER (Eye-behavior-aided Multimodal Emotion Recognition) Dataset: This dataset is novel in its inclusion of eye behaviors alongside facial expressions. It was collected from 121 participants who watched emotion-inducing videos. It contains:
  • Facial expression videos
    • Eye movement sequences (gaze coordinates, directions, pupil diameter, etc.)
    • Emotion-related eye fixation maps
    • Both ER (true emotion) and FER (facial expression) annotations, including 3-class (positive, negative, neutral), 7-class (happiness, sadness, fear, surprise, disgust, anger, neutral), and continuous valence/arousal ratings.

The dataset was collected using a stimulus material-induced spontaneous emotion generation approach.

Key Insights and Ideas

  • Eye behaviors are crucial for bridging the emotion gap: The study strongly suggests that eye movements and fixations provide critical information that can disambiguate facial expressions and reveal true emotions, especially when facial expressions are misleading.
  • Facial expressions and eye behaviors are mutually reinforcing: The research shows that integrating eye behavior data not only improves ER but also enhances FER performance, indicating a synergistic relationship.
  • The "emotion gap" is a significant challenge: The EMER dataset and analysis clearly demonstrate a noticeable discrepancy between perceived facial expressions and genuine emotions, highlighting the need for multimodal approaches.
  • Adversarial learning is effective for modality decoupling: The MAFD module's ability to disentangle modality-invariant and modality-specific features is key to handling the emotion gap and leveraging diverse modalities.

Key Results

  • State-of-the-art performance on EMER: The proposed EMERT model achieved state-of-the-art results on the EMER dataset for both ER and FER tasks across various metrics (WAR, UAR, F1 for classification; MAE, MSE, RMSE for regression).
  • For 3-class ER, EMERT achieved a WAR of 59.28%.
  • For 7-class ER, EMERT achieved a WAR of 33.92%.
  • For valence and arousal regression, EMERT achieved the best performance with the lowest MAE and MSE.
  • Significant improvement from eye behaviors: Ablation studies showed that incorporating eye movement data led to the highest performance improvements for both ER and FER tasks.
  • Robustness to noise: EMERT demonstrated superior robustness to noisy data compared to existing multimodal methods.
  • High annotation reliability: The proposed ALA method for FER annotation achieved a high Cronbach's Alpha of 0.978, indicating excellent consistency.

Limitations and Future Work

  • Limitations: The paper acknowledges that while the EMER dataset is large, further expansion with more participants and diverse demographics could be beneficial. The current FER annotation relies on machine auto-annotation combined with expert review, which might still have some inherent biases.
  • Future Work: The authors plan to extend the EMER database and release it to encourage further research. Future directions include:
  • False expression detection.
  • Emotion gap analysis between FER and ER.
  • Micro-expression detection.
  • Emotion intensity estimation.
  • Gaze-attention identification.
  • Exploring other related areas.

Summary updated

Abstract

Emotion Recognition (ER) is the process of identifying human emotions from given data. Currently, the field heavily relies on facial expression recognition (FER) because facial expressions contain rich emotional cues. However, it is important to note that facial expressions may not always precisely reflect genuine emotions and FER-based results may yield misleading ER. To understand and bridge this gap between FER and ER, we introduce eye behaviors as an important emotional cues for the creation of a new Eye-behavior-aided Multimodal Emotion Recognition (EMER) dataset. Different from existing multimodal ER datasets, the EMER dataset employs a stimulus material-induced spontaneous emotion generation method to integrate non-invasive eye behavior data, like eye movements and eye fixation maps, with facial videos, aiming to obtain natural and accurate human emotions. Notably, for the first time, we provide annotations for both ER and FER in the EMER, enabling a comprehensive analysis to better illustrate the gap between both tasks. Furthermore, we specifically design a new EMERT architecture to concurrently enhance performance in both ER and FER by efficiently identifying and bridging the emotion gap between the two.Specifically, our EMERT employs modality-adversarial feature decoupling and multi-task Transformer to augment the modeling of eye behaviors, thus providing an effective complement to facial expressions. In the experiment, we introduce seven multimodal benchmark protocols for a variety of comprehensive evaluations of the EMER dataset. The results show that the EMERT outperforms other state-of-the-art multimodal methods by a great margin, revealing the importance of modeling eye behaviors for robust ER. To sum up, we provide a comprehensive analysis of the importance of eye behaviors in ER, advancing the study on addressing the gap between FER and ER for more robust ER performance.
BibTeX
@misc{chen2024smile,
  author = {Zhe Chen},
  title = {{Smile upon the Face but Sadness in the Eyes: Emotion Recognition based on Facial Expressions and Eye Behaviors}},
  year = {2024},
  eprint = {2411.05879},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.05879},
}
Figure