Human-Centered Emotion Recognition in Animated GIFs

Zhengyuan Yang, Yixuan Zhang, Jiebo Luo

Abstract

Animated GIFs have emerged as a primary medium for emotional expression in digital communication, characterized by human-centric content and high temporal density. This paper presents the Keypoint Attended Visual Attention Network (KAVAN), a novel framework designed to capture these unique properties. KAVAN utilizes estimated facial keypoints to guide a soft attention mechanism and employs a Hierarchical Segment LSTM (HS-LSTM) to model short-term temporal dynamics. By integrating multi-task learning across 17 emotion categories and four affective quadrants, KAVAN achieves state-of-the-art performance in predicting emotional intensity.

Introduction

While automated emotion recognition in video has matured, animated GIFs present distinct challenges. Unlike standard videos, GIFs are typically short, looping, and lack "filler" frames, making every frame critical for context. Furthermore, they are predominantly human-centric, often focusing on facial expressions to convey sentiment. Traditional models often treat GIFs as generic video sequences, failing to leverage these inherent structural and content-based biases.

Methodology

The KAVAN architecture addresses these challenges through three primary innovations:

Facial Soft Attention Module

To prioritize facial regions, the model incorporates a soft attention mechanism. Using OpenPose to generate initial keypoint heatmaps, the network is trained to predict facial region masks. These masks, weighted by the estimator's confidence, serve as attention weights for a ResNet-50 feature extractor. This approach ensures robustness against noisy or missing keypoints, common in low-resolution GIFs or stylized cartoon content.

Hierarchical Segment LSTM (HS-LSTM)

Standard LSTMs often struggle with the dense information in GIFs, potentially "forgetting" early cues. KAVAN utilizes an HS-LSTM that partitions the GIF into segments. The model first learns coarse representations for each segment before hierarchically refining them into a global temporal understanding, preserving the integrity of the entire short sequence.

Multi-Task Learning (MTL)

The model is trained on two simultaneous tasks: predicting the intensity of 17 specific emotions (a regression task) and categorizing the GIF into one of four quadrants based on the psychological circumplex affect model (Arousal-Pleasure, Calm-Pleasure, Arousal-Misery, and Calm-Misery). This hierarchical approach provides a strong prior for the model, reducing significant errors by first identifying the broad affective state before refining the specific emotional intensity.

Dataset and Evaluation

The study utilizes the MIT GIFGIF Dataset, comprising 6,119 GIFs. Emotion labels were derived from over 3.2 million pairwise user comparisons, converted into continuous 17-dimensional intensity scores (ranging from -1 to 1) using the TrueSkill algorithm.

Figure 1: A GIF sequence illustrating the transition of facial expressions and the resulting 17-dimensional emotion intensity profile, categorized by the circumplex affect model.

Results and Discussion

KAVAN demonstrates superior performance across both classification and regression tasks:

  • Classification Accuracy: The model achieved 68.27% on the 4-class coarse categorization task, significantly outperforming the ResNet-50 + LSTM baseline of 61.47%.
  • Regression Performance: KAVAN reached a Normalized Mean Squared Error (nMSE) of 0.9863, surpassing the previous state-of-the-art (1.0000).

Qualitative analysis reveals that the soft attention mechanism successfully generalizes to cartoon characters, where traditional pose estimators often fail. By learning to predict attention masks from visual features rather than relying solely on raw coordinates, the model maintains high interpretability and accuracy across diverse visual styles.

Conclusion

KAVAN effectively addresses the unique properties of animated GIFs by combining human-centered attention with hierarchical temporal modeling. While the reliance on external pose estimation during training adds computational overhead, the resulting model is robust and interpretable. Future research may explore the inclusion of multimodal data and the application of these attention mechanisms to broader video emotion recognition tasks.

Summary updated

BibTeX
@inproceedings{yang2019human,
  author = {Zhengyuan Yang and Yixuan Zhang and Jiebo Luo},
  title = {{Human-Centered Emotion Recognition in Animated GIFs}},
  booktitle = {ICME},
  year = {2019},
  doi = {10.1109/icme.2019.00191},
}
Figure