Is Dynamic Rumor Detection on social media Viable? An Unsupervised Perspective

Chahat Raj, Priyanka MeelarXiv 2021 · OpenReview

TL;DR

This paper introduces an unsupervised framework for early detection of online rumors by leveraging content and social features of tweets and applying clustering techniques. The proposed method aims to overcome the limitations of supervised learning, which requires extensive labeled data, by identifying rumors as soon as they appear, outperforming existing baselines and some supervised methods.

What Problem Does This Paper Address?

The proliferation of online rumors on social media poses a significant challenge, as people increasingly rely on these platforms for information but are vulnerable to misinformation. Existing rumor detection methods often rely on supervised learning, which demands large, labeled datasets that are difficult to obtain in dynamic online environments. This makes early detection of rumors, as they emerge, a particularly challenging task. The paper addresses this gap by proposing an unsupervised approach that can identify rumors without prior labeling, enabling their detection at the earliest possible stage.

Methods

The core of the proposed methodology is an unsupervised classification framework. The authors extract two sets of features from tweets: content features (e.g., word count, presence of question marks, capital letter ratio) and social features (e.g., favorite count, retweet count, user's follower count, time difference between tweet and account creation). These features are then normalized and subjected to Principal Component Analysis (PCA) for dimensionality reduction, reducing the feature set to two principal components.

The reduced feature set is then fed into four different unsupervised clustering algorithms:

  • RBF Spectral Clustering
  • NN Spectral Clustering
  • K-Means Clustering
  • Fuzzy C-Means Clustering

The goal is to group tweets into two distinct clusters, representing rumors and non-rumors, based on their inherent characteristics. The paper emphasizes that this approach focuses on distinguishing rumors from non-rumors, not on verifying the truthfulness of a rumor.

Datasets

The paper uses the PHEME dataset, which is described as a large-scale rumor classification dataset. It consists of tweets from nine breaking-news events, with a total of 6425 tweets, of which 2402 are labeled as rumors and 4023 as non-rumors. The dataset is noted to be imbalanced, with an unequal number of rumors and non-rumors across events.

Key Insights and Ideas

  • Unsupervised Learning for Early Detection: The paper highlights the critical need for unsupervised methods in dynamic online environments where labeled data is scarce, enabling the detection of rumors as they emerge.
  • Synergy of Content and Social Features: The research demonstrates that combining content and social features of tweets leads to superior rumor detection performance compared to using either feature set alone.
  • PCA for Dimensionality Reduction: Using PCA effectively reduces the complexity of the feature space, allowing clustering algorithms to perform more efficiently and accurately.
  • Clustering as a Rumor Detection Mechanism: The study validates the effectiveness of clustering algorithms in separating rumors from non-rumors based on their inherent characteristics, even without explicit labels.

Key Results

The proposed unsupervised framework achieved competitive results across various events in the PHEME dataset.

  • For the Charlie Hebdo event, RBF spectral clustering achieved 77.97% accuracy using social features and a combination of content and social features.
  • The Germanwings Crash event saw the highest accuracy of 99.36% using Fuzzy C-means with content features.
  • For the Ottawa Shooting event, the highest accuracy was 97.19% using Fuzzy C-means with content features.
  • Compared to existing baselines, the proposed methodology demonstrated an improvement of approximately ~25-30% in F1-scores on several datasets. For instance, for the Charlie Hebdo dataset, the maximum F1-score was 87.62%, which is 11% higher than the baseline TextCNN.

Limitations and Future Work

  • Dataset Imbalance: The PHEME dataset, while large-scale, is imbalanced, which can affect the calculation of certain metrics like precision and recall. The authors acknowledge this challenge and focus on F1-score for comparison.
  • Focus on Distinction, Not Veracity: The current framework distinguishes between rumors and non-rumors but does not perform veracity analysis (i.e., determining if a rumor is true, false, or unverified).
  • Future Directions:
    • Reducing the number of feature attributes required for effective rumor detection.
  • Developing fine-grained classification to identify the stance of identified rumors.
  • Optimizing clustering algorithms to better generalize social media attributes.
  • Further research into early rumor detection to dynamically verify online posts.

Summary updated

Abstract

With the growing popularity and ease of access to the internet, the problem of online rumors is escalating. People are relying on social media to gain information readily but fall prey to false information. There is a lack of credibility assessment techniques for online posts to identify rumors as soon as they arrive. Existing studies have formulated several mechanisms to combat online rumors by developing machine learning and deep learning algorithms. The literature so far provides supervised frameworks for rumor classification that rely on huge training datasets. However, in the online scenario where supervised learning is exigent, dynamic rumor identification becomes difficult. Early detection of online rumors is a challenging task, and studies relating to them are relatively few. It is the need of the hour to identify rumors as soon as they appear online. This work proposes a novel framework for unsupervised rumor detection that relies on an online post's content and social features using state-of-the-art clustering techniques. The proposed architecture outperforms several existing baselines and performs better than several supervised techniques. The proposed method, being lightweight, simple, and robust, offers the suitability of being adopted as a tool for online rumor identification.
BibTeX
@misc{raj2021is,
  author = {Chahat Raj and Priyanka Meel},
  title = {{Is Dynamic Rumor Detection on social media Viable? An Unsupervised Perspective}},
  year = {2021},
  eprint = {2111.11982},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.11982},
}
Figure