It's about Time: Rethinking Evaluation on Rumor Detection Benchmarks using Chronological Splits

Yida Mu, Kalina Bontcheva, Nikolaos Aletras

Abstract

New events emerge over time, influencing the topics of rumors on social media. Current rumor detection benchmarks typically use random splits for training, development, and test sets, which often results in topical overlaps. Consequently, models trained on random splits may fail to generalize to previously unseen topics due to temporal concept drift. This paper provides a re-evaluation of classification models across four popular rumor detection benchmarks using chronological instead of random splits. Our results demonstrate that random splits significantly overestimate predictive performance, and we suggest that chronological splits should be the standard for minimizing topical overlaps in future evaluations.

Introduction

Rumor detection is a critical task in maintaining the integrity of social media information. Because rumors are inextricably linked to real-world events—such as elections, natural disasters, or celebrity news—they are inherently temporal. However, the standard practice in the research community has been to evaluate detection models using random data splits (e.g., 5-fold cross-validation) on benchmarks like Twitter 15, Twitter 16, and Weibo.

This paper argues that random splitting is a fundamental flaw in current evaluation methodologies. By ignoring the temporal dimension, researchers inadvertently allow models to "cheat" by training on data from the same events they are asked to predict in the test set. This paper investigates the impact of temporal concept drift—the phenomenon where the distribution of topics and language changes over time—and proposes a more rigorous chronological evaluation framework.

The Problem: Topical Overlap and Information Leakage

In a real-world scenario, a rumor detection model must identify rumors about events that have not yet occurred at the time of training. When a dataset is split randomly, tweets regarding a specific event (e.g., a specific protest or a celebrity's death) are distributed across both the training and testing sets.

The authors identified a significant "Information Leakage" trap: in the Twitter 16 dataset, approximately 30% of the test set consisted of posts highly similar to those in the training set under random splitting. Under chronological splits, this leakage drops to nearly zero. This suggests that high performance on random splits is often a result of the model memorizing specific event-based keywords rather than learning the underlying characteristics of rumor-mongering behavior.

Methodology

To quantify the performance gap, the researchers re-evaluated several state-of-the-art models using three splitting strategies:

  1. Random Splits (Baseline): The standard 5-fold cross-validation used in existing literature.
  2. Standard Chronological Splits: Sorting all posts by timestamp and using the first 70% for training, 10% for development, and the final 20% for testing.
  3. Stratified Chronological Splits: Splitting rumors and non-rumors separately by time before merging them. This ensures a balanced class distribution in the test set, preventing issues caused by sudden bursts of specific labels at the end of a dataset's timeframe.

These strategies were tested against three model architectures:

  • Logistic Regression (LR): A baseline using TF-IDF features.
  • BERT: The standard pre-trained transformer model.
  • BERT+: Domain-specific models, including BERTweet (for English) and ERNIE (for Chinese).

Experimental Results

The study utilized four major benchmarks: Twitter 15, Twitter 16, PHEME, and Weibo. The quantitative drop in performance when moving from random to chronological splits was stark across all datasets.

Performance Degradation

  • Twitter 16: F1-scores plummeted from approximately 90% to 47% for both BERT and BERTweet.
  • Twitter 15: F1-scores dropped from roughly 85% to 55%.
  • PHEME & Weibo: These larger datasets also saw significant declines, with performance dropping by 10-15%.

The massive performance hit on smaller datasets (Twitter 15/16) suggests these benchmarks are particularly vulnerable to event memorization. Even advanced BERT-style models were not immune to temporal drift; while they still outperformed simple Logistic Regression, their ability to generalize to "emerging" rumors—topics not present in the training set—was severely limited.

Discussion and Future Work

The findings indicate that current literature significantly overestimates the readiness of rumor detection models for live environments. Models that appear to be "superstars" on paper often fail when faced with the chronological reality of social media.

Limitations

While this study highlights a critical evaluation flaw, it is limited to text-based analysis on Twitter and Weibo. Future research should investigate whether incorporating user metadata, social graph propagation patterns, or multi-modal data can help mitigate the impact of temporal concept drift.

Conclusion

To ensure that rumor detection research translates to real-world impact, the field must move away from random data splits. Evaluating models chronologically provides a more honest assessment of their ability to handle the ever-changing landscape of online misinformation. Researchers are encouraged to adopt stratified chronological splits as the new standard for rumor detection benchmarks.

Summary updated

Abstract

New events emerge over time influencing the topics of rumors in social media. Current rumor detection benchmarks use random splits as training, development and test sets which typically results in topical overlaps. Consequently, models trained on random splits may not perform well on rumor classification on previously unseen topics due to the temporal concept drift. In this paper, we provide a re-evaluation of classification models on four popular rumor detection benchmarks considering chronological instead of random splits. Our experimental results show that the use of random splits can significantly overestimate predictive performance across all datasets and models. Therefore, we suggest that rumor detection models should always be evaluated using chronological splits for minimizing topical overlaps.
BibTeX
@inproceedings{mu2023it,
  author = {Yida Mu and Kalina Bontcheva and Nikolaos Aletras},
  title = {{It's about Time: Rethinking Evaluation on Rumor Detection Benchmarks using Chronological Splits}},
  booktitle = {EACL (Findings)},
  year = {2023},
  url = {https://openreview.net/forum?id=6PCfR6WCpYK},
}
Figure