Scalable and Generalizable Social Bot Detection through Data Selection

Kai-Cheng Yang, Onur Varol, Pik-Mai Hui, Filippo Menczer

Abstract

Social bot detection systems often struggle with real-time scalability and generalizing across diverse bot types. This study introduces a detection framework utilizing only 20 user-profile metadata features, enabling the processing of nearly 900 million tweets per day. By systematically evaluating 247 combinations of 14 datasets, the authors demonstrate that strategic data selection—rather than simply aggregating all available data—is key to robust cross-domain performance. The resulting model achieves high AUC scores on unseen datasets, outperforming traditional benchmarks.

Introduction

The proliferation of social bots necessitates detection tools that are both fast and adaptable. Current state-of-the-art methods often rely on "rich" features, such as tweet history or network structures, which are slow to retrieve via APIs and limit real-time monitoring. Furthermore, models often fail to generalize when encountering bot types not present in their training data (e.g., a model trained on 2011 spambots failing on 2018 political bots). This paper addresses these hurdles through lean feature engineering and optimized training data selection.

Methods

The framework employs a Random Forest classifier based on 20 features derived exclusively from Twitter "user objects" (metadata). These include 5 raw counts (e.g., followers, favorites) and 15 derived metrics, such as account growth rates and a "screen name likelihood" score using bigram frequencies.

SHAP values illustrating the impact of the 20 profile features on model output.

To understand the model's decision-making, SHAP (SHapley Additive exPlanations) was used to interpret feature importance. As shown in Figure 6, features like favourites_count and followers_count significantly influence the classification, with specific patterns distinguishing "human-like" from "bot-like" behavior.

The study utilized 14 datasets (94,000 bots, 43,000 humans), including benchmarks like caverlee and cresci-17, alongside new sets like midterm-18 and botwiki.

Key Results

A central finding is the "Less is More" paradox: models trained on a carefully selected subset of data (e.g., model M196) often outperform those trained on the entire aggregate (M246). This suggests that certain datasets introduce noise or highly specific patterns that hinder generalization.

Heatmap showing cross-dataset performance (AUC) for models trained and tested on different datasets.

Figure 2 illustrates the cross-domain performance across various training and test sets. The heatmap reveals significant asymmetry; for instance, a model trained on botwiki-verified performs well on vendor-verified, but the reverse is less effective. This highlights the importance of "separability" in training data.

The optimized model (M196) achieved an AUC of 0.99 on holdout sets like botwiki and midterm-18. Its efficiency allows for the evaluation of 900 million tweets per day on a single CPU, a 200x speedup over tools like Botometer.

Discussion and Limitations

While highly effective for individual account detection, the model struggles with coordinated campaigns where accounts appear human in isolation but act in sync (e.g., cresci-stock). Future work may involve instance-level data selection to further refine training sets and mitigate adversarial evasion by sophisticated bot operators who "groom" profile metadata to appear more human.

Summary updated

Abstract

Efficient and reliable social bot classification is crucial for detecting information manipulation on social media. Despite rapid development, state-of-the-art bot detection models still face generalization and scalability challenges, which greatly limit their applications. In this paper we propose a framework that uses minimal account metadata, enabling efficient analysis that scales up to handle the full stream of public tweets of Twitter in real time. To ensure model accuracy, we build a rich collection of labeled datasets for training and validation. We deploy a strict validation system so that model performance on unseen datasets is also optimized, in addition to traditional cross-validation. We find that strategically selecting a subset of training data yields better model accuracy and generalization than exhaustively training on all available data. Thanks to the simplicity of the proposed model, its logic can be interpreted to provide insights into social bot characteristics.
BibTeX
@inproceedings{yang2020scalable,
  author = {Kai-Cheng Yang and Onur Varol and Pik-Mai Hui and Filippo Menczer},
  title = {{Scalable and Generalizable Social Bot Detection through Data Selection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i01.5460},
}
Figure