It's about Time: Rethinking Evaluation on Rumor Detection Benchmarks using Chronological Splits

Yida Mu, Kalina Bontcheva, Nikolaos Aletras

TL;DR

Current rumor detection models are often evaluated using random data splits, which can lead to inflated performance metrics due to temporal concept drift. This paper demonstrates that chronological splits, where models are trained on older data and tested on newer data, reveal a significant drop in performance, highlighting the need for more realistic evaluation methods for real-world rumor detection systems.

What Problem Does This Paper Address?

The spread of misinformation and rumors on social media is a significant challenge. Automatic rumor detection systems aim to identify and flag these false narratives early on. However, the effectiveness of these systems is typically evaluated using benchmarks that split data randomly into training, validation, and test sets. This random splitting can lead to "topical overlap," where similar or identical rumors appear in both training and testing data. This is problematic because real-world events and topics evolve over time – a phenomenon known as "temporal concept drift." Models trained on random splits might appear to perform well because they've implicitly learned from similar content, but they may fail when faced with genuinely new, emerging rumors. This paper addresses the critical need to re-evaluate existing rumor detection benchmarks and models using chronological splits to better understand their true performance in dynamic, real-world scenarios.

Methods

The authors re-evaluate four popular rumor detection benchmarks using different data splitting strategies:

  • Standard Chronological Splits: Data is sorted by time, and then split into training (earliest 70%), development (next 10%), and testing (latest 20%) sets, ensuring no temporal overlap between sets. This is applied to datasets where temporal order is clear.
  • Stratified Chronological Splits: For datasets where temporal order might not be as straightforward or where there's no temporal overlap between rumors and non-rumors (like Twitter 16 and Weibo), this method splits rumors and non-rumors separately in chronological order before merging them into the final train, dev, and test sets. This aims to maintain temporal separation for each class.
  • Random Splits: This is the standard approach, using 5-fold cross-validation where data is randomly shuffled and split while preserving the original class distribution. This serves as the baseline for comparison.
  • Leave-One-Out (LOO) Splits: Used for reference on the PHEME dataset, this method involves leaving out one event at a time for testing.

The paper evaluates three types of models:

  • Logistic Regression (LR): A baseline classifier using TF-IDF weighted BOW features.
  • BERT: A pre-trained transformer model fine-tuned for the task.
  • BERT+ (BERTweet and ERNIE): Domain-specific pre-trained models (BERTweet for English social media, ERNIE for Chinese) fine-tuned similarly to BERT.

Datasets

The study uses four widely-used rumor detection benchmarks:

  • Twitter 15: Contains 1,490 tweets across four categories (Non-rumor, False Rumor, True Rumor, Unverified Rumor).
  • Twitter 16: Contains 818 tweets, also with four categories.
  • PHEME: Features 5,802 verified tweets from 9 real-world breaking news events, labeled as Rumor or Non-Rumor.
  • Weibo: A Chinese dataset with 4,664 verified posts, including rumors debunked by the Weibo Rumor Debunk Platform and non-rumors.

The authors note that while some benchmarks are suitable for standard chronological splits, others require the stratified approach due to their data characteristics.

Key Insights and Ideas

  • Random Splits Overestimate Performance: The most significant finding is that random splits consistently and significantly overestimate the performance of rumor detection models compared to chronological splits. This suggests that current benchmarks might be giving a misleadingly optimistic view of model capabilities.
  • Temporal Concept Drift is Real and Impactful: The study confirms that temporal concept drift is a major factor in rumor detection. As new events emerge, the nature of rumors changes, and models trained on past data struggle with novel topics.
  • Chronological Splits are Crucial for Real-World Evaluation: To accurately assess how well a rumor detection system will perform in practice, especially in identifying emerging rumors, chronological evaluation is essential.
  • Dataset Size Matters: The impact of temporal drift appears to be more pronounced on smaller datasets (like Twitter 15 and 16) than on larger ones (PHEME and Weibo), suggesting that smaller benchmarks are more susceptible to performance inflation from random splits.
  • Pre-trained Models Still Outperform: Even with chronological splits, pre-trained language models like BERT and BERT+ significantly outperform logistic regression, indicating their robustness. However, their performance also drops compared to random splits, suggesting potential information leakage from their pre-training data.

Key Results

  • Significant Performance Drops: Across all models and datasets, chronological splits resulted in substantially lower performance metrics (Precision, Recall, F1) compared to random splits. For instance, on Twitter 15, LR performance dropped from around 85% F1 with random splits to around 41% F1 with stratified chronological splits.
  • BERT and BERT+ Show Similar Trends: While these advanced models generally performed better, their performance also dropped significantly under chronological splits, confirming the issue of temporal concept drift.
  • Stratified vs. Standard Chronological: The study found that both standard and stratified chronological splits yielded similar model performance, suggesting that temporal drift affects specific classes rather than the entire dataset uniformly.

Limitations and Future Work

  • Limited Languages and Platforms: The study re-evaluated four benchmarks in English and Chinese across Twitter and Weibo. Further investigation into rumor detection datasets in other languages and platforms is needed.
  • Error Analysis: The authors performed an error analysis, noting that most errors occur with emerging or different topics compared to the training set, reinforcing the temporal drift issue. Wrong predictions are often posts similar to those in the training set when using chronological splits, indicating that the model is correctly identifying known patterns but struggling with novelty.
  • Future Directions: The authors plan to explore the impact of temporal concept drift on other NLP tasks, such as detecting user reactions to untrustworthy posts on social media.

Summary updated

Abstract

New events emerge over time influencing the topics of rumors in social media. Current rumor detection benchmarks use random splits as training, development and test sets which typically results in topical overlaps. Consequently, models trained on random splits may not perform well on rumor classification on previously unseen topics due to the temporal concept drift. In this paper, we provide a re-evaluation of classification models on four popular rumor detection benchmarks considering chronological instead of random splits. Our experimental results show that the use of random splits can significantly overestimate predictive performance across all datasets and models. Therefore, we suggest that rumor detection models should always be evaluated using chronological splits for minimizing topical overlaps.
BibTeX
@inproceedings{mu2023it,
  author = {Yida Mu and Kalina Bontcheva and Nikolaos Aletras},
  title = {{It's about Time: Rethinking Evaluation on Rumor Detection Benchmarks using Chronological Splits}},
  booktitle = {EACL (Findings)},
  year = {2023},
  url = {https://openreview.net/forum?id=6PCfR6WCpYK},
}
Figure