Happy Dance, Slow Clap: Using Reaction GIFs to Predict Induced Affect on Twitter

Boaz Shmueli, Soumya Ray, Lun-Wei KuACL 2021

TL;DR

This paper introduces ReactionGIF, a novel dataset of 30,000 Twitter interactions paired with reaction GIFs, enabling the prediction of induced emotions. They developed an automated method to collect this data and demonstrated its utility by creating baselines for predicting reaction categories, sentiment, and fine-grained emotions.

What Problem Does This Paper Address?

Understanding and predicting human emotions is crucial for many NLP applications, from dialogue systems to content moderation. However, existing emotion-labeled datasets primarily focus on perceived emotions (what emotions a text expresses) rather than induced emotions (what emotions a text evokes in the reader). Collecting induced emotion data is challenging, often requiring expensive and unreliable methods like physiological measurements or self-reporting. While some prior work has used emojis for distant supervision, their limited range and ambiguity hinder accurate emotion prediction. This paper tackles the scarcity of induced emotion data by leveraging the rich, expressive nature of reaction GIFs used in online conversations.

Methods

The authors propose a novel, fully-automated method for collecting induced emotion data using reaction GIFs. The core of their approach involves:

  1. GIF Dictionary Construction: They scraped Twitter to build a dictionary of 4300 GIFs, mapping each GIF to its pre-defined reaction category (e.g., "hug," "applause"). This leverages Twitter's two-step GIF selection process: users first choose a category, then select a GIF from that category.
  2. Reaction Category Inference: For a given text-GIF pair (t, g), they infer the reaction category of the GIF by looking it up in their dictionary. If a GIF appears in multiple categories, they prioritize the one where it appears earlier in the user-facing selection grid.
  3. Label Augmentation: Beyond the reaction category, they augment the dataset with:
  4. Sentiment Labels: Derived from hierarchical clustering of reaction categories, grouping them into positive and negative sentiment clusters.
  5. Emotion Labels: Mapped from reaction categories to a subset of 27 fine-grained emotions using a novel annotation process involving three human annotators.

Datasets

The primary contribution is the ReactionGIF dataset, a new benchmark comprising 30,000 English-language Twitter (text, GIF reaction) pairs. This dataset is unique in providing:

  • Induced emotion labels: Directly inferred from reaction GIFs.
  • Eliciting text: The original tweet that triggered the reaction.
  • Reaction categories: The specific category the GIF belongs to.
  • Augmented sentiment and emotion labels: Derived from the reaction categories.

This dataset is publicly available.

Key Insights and Ideas

  • Reaction GIFs as a Rich Source of Induced Affect: The paper convincingly argues that reaction GIFs, often used to express immediate emotional responses, are a powerful and underutilized source for capturing induced affect in online interactions.
  • Hierarchical Clustering Reveals Affective Relationships: The hierarchical clustering of reaction categories reveals intuitive groupings, such as "mic drop" to "smh" for negative sentiment and "hug" to "slow clap" for positive sentiment. This demonstrates that even seemingly simple reaction categories carry complex affective signals.
  • Induced vs. Perceived Emotion Distinction is Critical: The paper highlights the importance of distinguishing between perceived and induced emotions, emphasizing that induced emotions are more relevant for applications aiming to influence or respond to user affect.
  • Multilabel Emotion Prediction is Necessary: The 27-emotion taxonomy and the resulting multilabel classification task underscore the complexity of human emotional responses, which cannot always be captured by a single label.

Key Results

The paper establishes baselines for three tasks using the ReactionGIF dataset:

  • Affective Reaction Prediction: RoBERTa achieved a weight-averaged F1-score of 23.9% for predicting the reaction category.
  • Induced Sentiment Prediction: RoBERTa achieved 70.0% accuracy and 69.8% F1-score for binary sentiment classification.
  • Induced Emotion Prediction: RoBERTa achieved a Label Ranking Average Precision (LRAP) of 0.596 for the 27-emotion multilabel classification task.

These results demonstrate the viability of the ReactionGIF dataset and the proposed methods for predicting induced affect.

Limitations and Future Work

Limitations: * GIF Dictionary Size: The current GIF dictionary of 4300 GIFs is considered a limitation, and a larger dictionary is suggested to improve accuracy. * Category Overlap: The inherent overlap between reaction categories (e.g., "thumbs up" and "ok") presents a challenge for precise classification. * Long-Tail Distribution: The category distribution exhibits a long tail, meaning many categories have very few samples, making them harder to predict.

Future Work: * Expanding the GIF Dictionary: Incorporating more GIFs to cover a wider range of affective expressions. * Improving Mapping Accuracy: Developing more precise methods for mapping reactions to emotions. * Exploring Other Platforms: Generalizing the method to capture data from other social media and instant messaging platforms. * Downstream Applications: Applying the dataset and models to tasks like GIF recommendation, dialogue systems, and multimodal emotion detection. * Addressing Ethical Concerns: Investigating the potential risks of inducing affective responses and ensuring responsible deployment.

Summary updated

BibTeX
@inproceedings{shmueli2021happy,
  author = {Boaz Shmueli and Soumya Ray and Lun-Wei Ku},
  title = {{Happy Dance, Slow Clap: Using Reaction GIFs to Predict Induced Affect on Twitter}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure