CLEARumor at SemEval-2019 Task 7: ConvoLving ELMo Against Rumors

Ipek Baris, Lukas Schmelzeisen, Steffen Staab

Abstract

This paper describes our submission to SemEval-2019 Task 7: RumourEval: Determining Rumor Veracity and Support for Rumors. We participated in both subtasks. The goal of subtask A is to classify the type of interaction between a rumorous social media post and a reply post as support, query, deny, or comment. The goal of subtask B is to predict the veracity of a given rumor. For subtask A, we implement a CNN-based neural architecture using ELMo embeddings of post text combined with auxiliary features and achieve a F1-score of 44.6%. For subtask B, we employ a MLP neural network leveraging our estimates for subtask A and achieve a F1-score of 30.1% (second place in the competition). We provide results and analysis of our system performance and present ablation experiments.

Introduction: The Challenge of Social Media Rumors

Social media is a double-edged sword: while it enables rapid information sharing, it also serves as a fertile ground for misinformation. Identifying whether a rumor is true, false, or unverified is incredibly difficult for automated systems because definitive "ground truth" evidence is rarely available at the moment a rumor begins to spread.

To address this, researchers utilize Stance Analysis. Rather than analyzing the rumor in isolation, systems examine how the crowd reacts. Are users supporting, denying, questioning, or simply commenting on the claim? This research addresses two specific subtasks from the SemEval-2019 RumourEval challenge:

  1. Subtask A (SDQC): Classifying individual replies in a thread as Support, Deny, Query, or Comment.
  2. Subtask B (Veracity): Predicting the truth value (True, False, or Unverified) of the source rumor based on the entire conversation thread.
Example of a rumor thread with stance labels (SDQC) and veracity label.

Methodology: The CLEARumor Architecture

The proposed architecture, CLEARumor (ConvoLving ELMo Against Rumors), utilizes a multi-stage pipeline to process social media text and metadata:

  • Contextual Embeddings: Unlike traditional word embeddings that provide static vectors, CLEARumor employs ELMo. ELMo analyzes the entire sentence to capture context-dependent meanings, which is vital for social media where sarcasm and slang are prevalent.
  • Stance Classifier (Subtask A): ELMo-embedded text is processed by a Convolutional Neural Network (CNN) using multiple filter sizes (kernels of 2 and 3) to capture local textual patterns. These features are concatenated with auxiliary features, such as user verification status, follower counts, and reply depth.
  • Veracity Predictor (Subtask B): This stage uses a Multilayer Perceptron (MLP) that leverages the outputs of Subtask A. By calculating the distribution of support, denial, and query stances across a thread, the model determines the rumor's likely veracity.
The CLEARumor architecture: ELMo embeddings followed by convolutional layers, global average pooling, and auxiliary feature integration.

Data and Experimental Setup

The study utilizes the RumourEval 2019 dataset, comprising labeled threads from Twitter and Reddit. The dataset includes 8,529 individual posts for stance classification and 456 rumor threads for veracity prediction. A significant challenge noted by the authors is class imbalance; for example, "Comments" dominate the replies, while "Denials" are relatively rare.

Key Results and Insights

  • Performance: The system achieved an F1-score of 30.1% in veracity prediction, earning second place in the official SemEval-2019 competition. For stance classification, the full CNN + Auxiliary model achieved a macro F1-score of 44.6%.
  • The Power of the Crowd: Veracity is best predicted by aggregating the stance of replies. A high frequency of "Deny" and "Query" labels strongly correlates with a rumor being flagged as False.
  • Regularization: Due to the small dataset size, the authors kept ELMo weights frozen to prevent overfitting, allowing the model to generalize better to unseen rumors.
  • Metadata Importance: Textual analysis alone is insufficient. Metadata features, such as the upvote-to-downvote ratio on Reddit, provided critical context that significantly improved accuracy.

Conclusion

While CLEARumor demonstrates the effectiveness of combining contextual embeddings with stance analysis, limitations remain regarding platform-specific feature availability and the small size of labeled rumor datasets. Future work will explore transformer-based models like BERT and investigate cross-lingual rumor detection capabilities.

Summary updated

Abstract

This paper describes our submission to SemEval-2019 Task 7: RumourEval: Determining Rumor Veracity and Support for Rumors. We participated in both subtasks. The goal of subtask A is to classify the type of interaction between a rumorous social media post and a reply post as support, query, deny, or comment. The goal of subtask B is to predict the veracity of a given rumor. For subtask A, we implement a CNN-based neural architecture using ELMo embeddings of post text combined with auxiliary features and achieve a F1-score of 44.6%. For subtask B, we employ a MLP neural network leveraging our estimates for subtask A and achieve a F1-score of 30.1% (second place in the competition). We provide results and analysis of our system performance and present ablation experiments.
BibTeX
@inproceedings{baris2019clearumor,
  author = {Ipek Baris and Lukas Schmelzeisen and Steffen Staab},
  title = {{CLEARumor at SemEval-2019 Task 7: ConvoLving ELMo Against Rumors}},
  booktitle = {SemEval @ NAACL},
  year = {2019},
  url = {https://openreview.net/forum?id=khrpwfOib3Y},
}
Figure