Call Attention to Rumors: Deep Attention Based Recurrent Neural Networks for Early Rumor Detection

Tong Chen, Xue Li, Hongzhi Yin, Jun Zhang

TL;DR

This paper introduces "CallAtRumors," a novel deep learning model that uses recurrent neural networks (RNNs) with a soft attention mechanism to detect rumors on social media early in their diffusion. By focusing on the most relevant parts of sequential posts, CallAtRumors significantly outperforms existing methods in both accuracy and speed, even with highly duplicated content.

What Problem Does This Paper Address?

Social media platforms are fertile ground for the rapid spread of rumors, which can have serious consequences for cybersecurity and social stability. Detecting these rumors early, before they gain widespread traction, is crucial. However, this "early rumor detection" is challenging due to several factors:

  • Data Volume and Velocity: Rumors spread through a continuous stream of posts, creating large, time-series datasets.
  • Textual Duplication: Users often forward or comment on existing posts with minor variations, leading to significant textual overlap that can confuse traditional models.
  • Dynamic Content: The nature of posts changes as a rumor evolves, from initial surprise to questioning and eventual debunking, making it hard to capture consistent features.
  • Manual Feature Engineering: Existing methods often rely on hand-crafted features (e.g., sentiment, user profiles, propagation patterns), which are time-consuming to develop and may not generalize well.

The paper aims to address these challenges by developing an automated, flexible model that can effectively learn from sequential social media data and identify rumors at their earliest stages.

Methods

The core of the proposed solution is a Deep Attention Based Recurrent Neural Network (RNN), named CallAtRumors.

  • Recurrent Neural Networks (RNNs): The paper leverages RNNs, specifically Long Short-Term Memory (LSTM) units, to process the sequential nature of social media posts. LSTMs are well-suited for capturing long-range dependencies in time-series data, which is essential for understanding how a rumor evolves.
  • Soft Attention Mechanism: This is the key innovation. The attention mechanism allows the model to dynamically weigh the importance of different words or phrases within each post. Instead of treating all parts of a post equally, it learns to focus on the most discriminative features that are indicative of a rumor. This is particularly useful for handling duplicated content, as the attention can highlight the subtle differences or key phrases that matter. The paper uses a deterministic soft attention, meaning it's differentiable and can be trained end-to-end using standard backpropagation.
  • TF-IDF Feature Representation: Posts are first converted into tf-idf vectors, a standard technique for representing the importance of words in a document relative to a corpus. This provides a numerical input for the RNN.
  • Event-Level Detection: The model operates at the event level, meaning it considers all posts related to a specific topic or event as a sequence, rather than trying to classify individual posts in isolation.

Datasets

The study utilizes two publicly available datasets:

  • Twitter: Originally collected by [17], this dataset contains events labeled as rumors or non-rumors. The authors note that some tweets were no longer available during their crawling, leading to a slight reduction in dataset size.
  • Sina Weibo: This dataset also contains labeled events.

The authors followed the methodology of [17] to ensure comparability, including manually gathering additional data to achieve a 1:1 ratio of rumors to non-rumors in both datasets.

Key Insights and Ideas

  • Attention is Crucial for Duplicated Content: The paper strongly suggests that the attention mechanism is vital for overcoming the challenge of highly duplicated content in social media. By learning to focus on specific, discriminative words, the model can extract meaningful signals even when many posts say similar things.
  • Early Detection Benefits from Focused Features: The ability of the attention mechanism to identify important features early on is key to achieving early rumor detection. It allows the model to make accurate predictions even with limited data, which is characteristic of the formative stages of a rumor.
  • RNNs + Attention is a Powerful Combination: The synergy between RNNs (for temporal context) and attention (for selective focus) creates a robust framework for analyzing sequential social media data.

Key Results

  • Superior Performance: CallAtRumors consistently outperformed state-of-the-art baselines (DT-Rank, LK-RBF, SVM-TS, ML-GRU, CERT) on both the Twitter and Weibo datasets.
  • On the Twitter dataset, CallAtRumors achieved a precision of 88.63%, recall of 85.71%, and F-measure of 0.8715.
  • On the Weibo dataset, it achieved a precision of 87.10%, recall of 86.34%, and F-measure of 0.8672.
  • Early Detection Advantage: Experiments on early detection showed that CallAtRumors significantly outperformed other methods when trained on only 10-60% of the data, demonstrating its effectiveness in identifying rumors at their nascent stages. For instance, it showed a 5% precision and 4% recall advantage over ML-GRU at the 10-20% data proportion.
  • Visualization of Attention: The paper includes visualizations showing how the attention mechanism assigns different degrees of importance to words in a post, highlighting how it focuses on relevant terms and can ignore less informative ones.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the performance of these methods can drop significantly when there isn't enough data with differentiable content in the early stages of a rumor.
  • While the model handles duplicated content well, the paper notes that "some Tweets are no longer available" when crawling datasets, which can affect data scale.
  • Future Work:
    • The authors suggest investigating the possibility of combining more complex features with their deep attention model.
  • They also propose exploring the modeling of propagation patterns of rumors as sequential inputs for RNNs to further improve detection accuracy.
  • Future work could also investigate efficiency improvements using hashing techniques over multi-level feature spaces.

Summary updated

Abstract

The proliferation of social media in communication and information dissemination has made it an ideal platform for spreading rumors. Automatically debunking rumors at their stage of diffusion is known as early rumor detection, which refers to dealing with sequential posts regarding disputed factual claims with certain variations and highly textual duplication over time. Thus, identifying trending rumors demands an efficient yet flexible model that is able to capture long-range dependencies among postings and produce distinct representations for the accurate early detection. However, it is a challenging task to apply conventional classification algorithms to rumor detection in earliness since they rely on hand-crafted features which require intensive manual efforts in the case of large amount of posts. This paper presents a deep attention model based on recurrent neural networks (RNNs) to selectively learn temporal representations of sequential posts for rumor identification. The proposed model delves soft-attention into the recurrence to simultaneously pool out distinct features with particular focus and produce hidden representations that capture contextual variations of relevant posts over time. Extensive experiments on real datasets collected from social media websites demonstrate that the deep attention based RNN model outperforms state-of-the-art baselines by detecting rumors more quickly and accurately than competitors.
BibTeX
@inproceedings{chen2018call,
  author = {Tong Chen and Xue Li and Hongzhi Yin and Jun Zhang},
  title = {{Call Attention to Rumors: Deep Attention Based Recurrent Neural Networks for Early Rumor Detection}},
  booktitle = {PAKDD (Workshops)},
  year = {2018},
  doi = {10.1007/978-3-030-04503-6\_4},
}
Figure