Adapting Emotion Detection to Analyze Influence Campaigns on Social Media

Ankita Bhaumik, Andy Bernhardt, Gregorios Katsios, Ning Sa, Tomek StrzalkowskiACL 2023

TL;DR

This paper introduces a generalized approach to emotion detection that can be effectively adapted to the political domain without extensive retraining. By leveraging hierarchical emotion groupings and a novel label transfer mechanism, the authors demonstrate improved zero-shot performance on analyzing influence campaigns on social media, particularly during elections.

What Problem Does This Paper Address?

Analyzing influence campaigns on social media is crucial, especially during significant events like elections or pandemics. Emotions play a central role in these campaigns, shaping public opinion and driving behavior. However, accurately detecting emotions in the political domain is challenging due to the lack of specialized, large-scale datasets and the nuanced, context-dependent nature of political emotions. Existing emotion detection models, often trained on general datasets with standard emotion labels (like Ekman's), struggle to capture the specific emotional expressions relevant to political discourse. This often necessitates costly and time-consuming manual annotation for new domains, highlighting the need for more adaptable and robust emotion detection systems.

Methods

The core of the proposed solution is an ensemble model that combines traditional sentiment analysis with fine-tuned emotion detection models. The novelty lies in its hierarchical label transfer mechanism.

Here's a breakdown:

  • Hierarchical Emotion Grouping: The authors first group standard emotion labels into broader categories relevant to political discourse. This is informed by psychological theories and observed co-occurrences of emotions in political contexts. For example, "anger, hate, contempt, disgust" form one group.
  • Ensemble of Pre-trained Models: They utilize pre-trained language models (like RoBERTa) fine-tuned on existing emotion datasets (CBET and GoEmotions) for six Ekman emotions and their subcategories (joy, sadness, fear, anger).
  • Sentiment Integration: A sentiment analysis model is also integrated into the ensemble, recognizing that sentiment often provides a crucial contextual layer for understanding political emotions.
  • Label Transfer Mechanism: This is the key innovation. Instead of directly mapping raw emotion scores, the system uses a weighted linear combination. This mapping considers:
  • The sentiment category (positive/negative) associated with a target emotion.
  • The relationship between Ekman emotions and their subcategories.
  • The co-occurrence of emotions and their relevance to the target political domain.
  • Zero-Shot and Few-Shot Adaptation: The approach is designed for zero-shot (no in-domain training data) and few-shot (limited in-domain data) scenarios. For few-shot learning, they explore optimizing classification thresholds and mapping weights using a small annotated validation set.

Datasets

The paper utilizes the following datasets:

  • Cleaned Balanced Emotional Tweets (CBET): An 81k tweet dataset for training models on standard Ekman emotions.
  • GoEmotions: A large corpus of 58k English Reddit comments with 27 fine-grained emotion labels. This dataset is used to train models for Ekman emotions and their subcategories.
  • 2017 French Presidential Election Dataset: This is the primary dataset used for evaluation. It's a collection of tweets from the French election, which the authors annotated with their custom emotion label set. This dataset is crucial for demonstrating the model's adaptability to a specialized domain.

Key Insights and Ideas

  • Political Emotions are Nuanced: Standard emotion labels are insufficient for political discourse. Emotions like "love" and "admiration" can be synonymous in a political context but distinct in general taxonomies, necessitating domain-specific adaptation.
  • Sentiment is Crucial: Integrating sentiment analysis significantly improves emotion detection performance in political contexts, underscoring the interplay between overall sentiment and specific emotional expressions.
  • Hierarchical Grouping and Label Transfer are Effective: The proposed method of grouping emotions and using a weighted label transfer mechanism allows for effective zero-shot adaptation, outperforming baselines that rely solely on standard definitions.
  • Low Inter-Rater Reliability Highlights the Challenge: The low inter-rater reliability (17% kappa) on the French election dataset annotations underscores the inherent difficulty and subjectivity in labeling political emotions, further justifying the need for robust, adaptable models.

Key Results

The paper demonstrates the effectiveness of their approach, particularly in zero-shot settings:

  • Superior Zero-Shot Performance: The proposed ensemble model achieved significantly higher F1 scores across most emotion labels compared to traditional zero-shot baselines (TE and SB) on the French election dataset. For example, "optimism/hope" reached an F1 of 0.30 with their method, compared to 0.22 for TE and 0.16 for SB.
  • Contribution of Sentiment: An ablation study showed that including the sentiment component in the ensemble improved scores, highlighting its importance.
  • Few-Shot Performance: While optimizing thresholds and mapping weights in a few-shot setting showed some improvement (F1 of 0.39 vs. 0.37 for zero-shot), performance was limited by inconsistencies in annotation between validation and test sets.

Limitations and Future Work

  • Multilingualism: The current approach relies on translation to English for multilingual tweets. Future work should explore using multilingual pre-trained models and non-English training data for a more language-agnostic emotion model.
  • In-Domain Data Quality: The performance drop in the few-shot setting due to annotation inconsistencies suggests that high-quality, in-domain validation data is critical for optimal adaptation.
  • Publicly Available Dataset: The authors aim to release a publicly available dataset annotated with political emotions to facilitate further research.
  • Human Evaluation: The authors recommend human evaluation of results obtained in unknown domains before deploying them for downstream analytics.

Summary updated

BibTeX
@inproceedings{bhaumik2023adapting,
  author = {Ankita Bhaumik and Andy Bernhardt and Gregorios Katsios and Ning Sa and Tomek Strzalkowski},
  title = {{Adapting Emotion Detection to Analyze Influence Campaigns on Social Media}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure