Joint Rumour Stance and Veracity Prediction

Anders Edelbo Lillie, Emil Refsgaard Middelboe, Leon DerczynskiNODALIDA 2019 · OpenReview

Abstract

The net is rife with rumours that spread through microblogs and social media. Not all the claims in these can be verified. However, recent work has shown that the stances alone that commenters take toward claims can be sufficiently good indicators of claim veracity, using e.g. an HMM that takes conversational stance sequences as the only input. Existing results are monolingual (English) and mono-platform (Twitter). This paper introduces a stance-annotated Reddit dataset for the Danish language, and describes various implementations of stance classification models. Of these, a Linear SVM provides predicts stance best, with 0.76 accuracy / 0.42 macro F1. Stance labels are then used to predict veracity across platforms and also across languages, training on conversations held in one language and using the model on conversations held in another. In our experiments, monolinugal scores reach stance-based veracity accuracy of 0.83 (F1 0.68); applying the model across languages predicts veracity of claims with an accuracy of 0.82 (F1 0.67). This demonstrates the surprising and powerful viability of transferring stance-based veracity prediction across languages.

Introduction

The rapid proliferation of misinformation on social media platforms necessitates automated verification systems. While traditional fact-checking relies on external knowledge bases, stance-based veracity prediction leverages the "wisdom of the crowd." By analyzing how users react to a claim—whether they support, deny, or query it—models can infer the truthfulness of a rumor without direct factual verification. While this approach has shown promise on English Twitter data, its applicability to other languages and more complex platforms like Reddit remains under-explored. This paper addresses this gap by introducing the Danish Stance-annotated Reddit Dataset (DAST) and evaluating cross-lingual veracity prediction.

Methodology

The researchers implemented a two-stage pipeline consisting of stance classification followed by veracity prediction.

Stance Classification

Individual comments were annotated using the SDQC scheme: Supporting, Denying, Querying, or Commenting. The study compared several models, including traditional machine learning (SVM, Logistic Regression) and deep learning (Branch-LSTM).

Handling Reddit Structure

Unlike the relatively flat structure of Twitter, Reddit features deeply nested conversations. To account for this, the authors tested three structural representations:

  • Branch-as-source (BAS): Each individual branch is a separate instance.
  • Top-level-comment-as-source (TCAS): Each top-level comment and its sub-thread is a unit.
  • Submission-as-source (SAS): The entire post and all its comments are treated as one instance.
Reddit conversation structure and branching

Veracity Prediction

A Hidden Markov Model (HMM) was employed to analyze the sequence of stances within a thread. The HMM treats the conversation as a sequence of states, where specific patterns of SDQC labels correlate with the veracity of the underlying rumor.

Dataset: DAST

The Danish Stance-annotated Reddit Dataset (DAST) comprises 3,007 annotated posts across 11 events, including the Peter Madsen case, 5G conspiracy theories, and the #MeToo movement. This dataset provides a critical benchmark for non-English stance analysis on a platform with complex conversational dynamics.

Results

Stance Classification Performance

Experimental results indicated that on the DAST dataset, traditional feature-engineered models outperformed deep learning approaches. A Linear SVM achieved the highest accuracy of 0.76, though macro $F_1$ scores remained modest due to the heavy imbalance toward the "Commenting" label.

Stance classification performance across different models

Veracity Prediction and Cross-Lingual Transfer

The HMM-based veracity prediction achieved a monolingual accuracy of 0.83 on Danish data. Most significantly, when the model was trained on English and German Twitter data (PHEME dataset) and tested on Danish Reddit data, it maintained an accuracy of 0.82. This suggests that the social signals of skepticism and support are consistent across different languages and social media platforms.

Conclusion

This work demonstrates that stance-based veracity prediction is a robust and portable method for rumor verification. The success of cross-lingual transfer indicates that human patterns of argumentation regarding rumors are remarkably universal. Future work will focus on addressing data imbalance and scaling the DAST dataset to better support deep learning architectures.

Summary updated

Abstract

The net is rife with rumours that spread through microblogs and social media. Not all the claims in these can be verified. However, recent work has shown that the stances alone that commenters take toward claims can be sufficiently good indicators of claim veracity, using e.g. an HMM that takes conversational stance sequences as the only input. Existing results are monolingual (English) and mono-platform (Twitter). This paper introduces a stance-annotated Reddit dataset for the Danish language, and describes various implementations of stance classification models. Of these, a Linear SVM provides predicts stance best, with 0.76 accuracy / 0.42 macro F1. Stance labels are then used to predict veracity across platforms and also across languages, training on conversations held in one language and using the model on conversations held in another. In our experiments, monolinugal scores reach stance-based veracity accuracy of 0.83 (F1 0.68); applying the model across languages predicts veracity of claims with an accuracy of 0.82 (F1 0.67). This demonstrates the surprising and powerful viability of transferring stance-based veracity prediction across languages.
BibTeX
@inproceedings{lillie2019joint,
  author = {Anders Edelbo Lillie and Emil Refsgaard Middelboe and Leon Derczynski},
  title = {{Joint Rumour Stance and Veracity Prediction}},
  booktitle = {NODALIDA},
  year = {2019},
  url = {https://openreview.net/forum?id=avJxLBTvpiE},
}
Figure