Intensity-Valued Emotions Help Stance Detection of Climate Change Twitter Data

Apoorva Upadhyaya, Marco Fisichella, Wolfgang NejdlIJCAI 2023 · DOI

TL;DR

This paper introduces SIMS, a novel multi-task framework that leverages emotion recognition and intensity prediction to improve stance detection on climate change-related tweets. By capturing the nuanced emotional landscape of online discourse, SIMS achieves state-of-the-art performance, offering a more robust way to understand public opinion on this critical issue.

What Problem Does This Paper Address?

The paper tackles the challenge of accurately identifying public attitudes towards climate change on social media, specifically Twitter. Climate change discussions are often polarized, fueled by misinformation and fake news, making it difficult to discern genuine stances from noise. Existing stance detection models often struggle with the subtle nuances of language, particularly when users express similar sentiments with varying emotional intensity. This can lead to misclassification, hindering efforts to understand and address public opinion on climate action. The authors highlight that previous work has either lacked efficient architectures or overlooked the crucial role of emotions and their intensity in shaping these online conversations.

Methods

The core of the proposed approach is the SIMS (Intensity-Valued Emotions Help Stance Detection) framework. SIMS is a multi-task learning model that integrates three key components:

  • Embedding Component: This module first uses an "Affect Words Extractor" to identify words in a tweet that influence emotions. These affect words, along with the tweet text, are then processed by a BERTweet-based embedding module. An "Emotion-Affect Inducer" then uses a convolution operation to inject emotional aspects into the tweet representation. Finally, the text and emotion-infused representations are averaged to create an "emotionally affected tweet" representation.
  • Feature Encoder: Three separate Bi-GRU (Bidirectional Gated Recurrent Unit) layers are used to encode the emotionally affected tweet representation for each of the three tasks: stance detection (SD), emotion recognition (ER), and emotion intensity prediction (EI).
  • Attention Module: This component extracts task-specific attention vectors for each of the three tasks. It then fuses these with a shared attention vector (derived from averaging the task-specific ones) using an integration module that employs absolute difference and element-wise product operations. This fusion aims to leverage both task-specific and common features.
  • Classification Layer: The final fused representation is passed through separate output channels for each task, using appropriate loss functions (categorical cross-entropy for SD, binary cross-entropy for ER, and mean squared error for EI).

The novelty lies in the explicit incorporation of emotion recognition and intensity prediction as auxiliary tasks, and the specific way emotions are injected and fused with text representations.

Datasets

The authors utilized a combination of datasets for their experiments:

  • Newly Curated Dataset: A custom dataset was created by collecting tweets using climate change-related hashtags and then manually annotating them for stance, emotions, and intensity.
  • Publicly Available Datasets:
    • ClimateICWSM-2023: A dataset of climate change tweets with believe and deny stances.
  • ClimateStance-2022: A benchmark dataset for climate change stance detection, including tweets supporting, opposing, or being ambiguous about climate change prevention.
  • SemEval-2016: A well-known benchmark dataset for stance detection across various targets, including climate change.

Key Insights and Ideas

  • Emotions as Crucial Signals: The study strongly suggests that emotions, and critically, their intensity, are vital for accurately determining stance on climate change. Deniers' tweets often exhibit higher levels of anger and disgust, while believers show more anticipation and trust.
  • Intensity Matters: Even when similar emotions are present, their intensity can differentiate between stances. For example, a "gentle" tone with lower intensity anger might be present in a believer's tweet, contrasting with a more aggressive expression from a denier.
  • Multi-Task Learning Synergy: Using emotion recognition and intensity prediction as auxiliary tasks significantly boosts the performance of the primary stance detection task, demonstrating the benefit of a holistic approach.
  • BERTweet's Advantage: Pre-training on tweets (BERTweet) provides a more efficient and effective embedding for this domain compared to general BERT embeddings.
  • Attention Fusion is Key: The integration of task-specific and shared attention vectors, rather than simple concatenation, leads to a substantial performance improvement, highlighting the importance of how features are combined.

Key Results

The SIMS model achieved state-of-the-art results across the evaluated datasets.

  • On their curated dataset, SIMS outperformed all baselines, achieving an F1 score of 91.84% when all three tasks (SD, ER, EI) were integrated.
  • The single-task model variant (BERTweet + AWE + TA) achieved an F1 score of 85.19%, demonstrating the foundational importance of emotion-aware features.
  • When comparing multi-task architectures, the SIMS model (EM+EAI+TA+SA) achieved an F1 score of 90.06% for the combined SD+ER+EI task, outperforming simpler combinations.
  • On the ClimateICWSM-2023 dataset, SIMS achieved an accuracy of 95.01% and an F1 score of 93.52%.
  • On the SemEval-2016 dataset, SIMS achieved an overall F1 score of 71.24%, showing good generalization across different targets.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Insufficient Context: In cases where tweet text is sparse, SIMS can misclassify ambiguous tweets. The presence of other modalities (like videos, which were mentioned in an example) could significantly improve accuracy.
  • Implicit/Hidden Stances: Tweets that express support for climate change but oppose government action are challenging to classify. The authors suggest that finer categorization of stance classes (e.g., "believer but against action") could improve performance.
  • Generalization to Other Topics: While the model shows good generalization on SemEval-2016, further work could explore its effectiveness on a wider range of topics beyond climate change.
  • Real-time Suitability: Detailed analysis of latency and prediction time for real-time implementation is noted as future work.
  • Exploring Other Modalities: Future research could focus on incorporating other modalities like images and videos to enrich the context for stance detection.

Summary updated

BibTeX
@inproceedings{upadhyaya2023intensity,
  author = {Apoorva Upadhyaya and Marco Fisichella and Wolfgang Nejdl},
  title = {{Intensity-Valued Emotions Help Stance Detection of Climate Change Twitter Data}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/693},
}
Figure