Abstract
Social bot detection systems often struggle with real-time scalability and generalizing across diverse bot types. This study introduces a detection framework utilizing only 20 user-profile metadata features, enabling the processing of nearly 900 million tweets per day. By systematically evaluating 247 combinations of 14 datasets, the authors demonstrate that strategic data selection—rather than simply aggregating all available data—is key to robust cross-domain performance. The resulting model achieves high AUC scores on unseen datasets, outperforming traditional benchmarks.
Introduction
The proliferation of social bots necessitates detection tools that are both fast and adaptable. Current state-of-the-art methods often rely on "rich" features, such as tweet history or network structures, which are slow to retrieve via APIs and limit real-time monitoring. Furthermore, models often fail to generalize when encountering bot types not present in their training data (e.g., a model trained on 2011 spambots failing on 2018 political bots). This paper addresses these hurdles through lean feature engineering and optimized training data selection.
Methods
The framework employs a Random Forest classifier based on 20 features derived exclusively from Twitter "user objects" (metadata). These include 5 raw counts (e.g., followers, favorites) and 15 derived metrics, such as account growth rates and a "screen name likelihood" score using bigram frequencies.
To understand the model's decision-making, SHAP (SHapley Additive exPlanations) was used to interpret feature importance. As shown in Figure 6, features like favourites_count and followers_count significantly influence the classification, with specific patterns distinguishing "human-like" from "bot-like" behavior.
The study utilized 14 datasets (94,000 bots, 43,000 humans), including benchmarks like caverlee and cresci-17, alongside new sets like midterm-18 and botwiki.
Key Results
A central finding is the "Less is More" paradox: models trained on a carefully selected subset of data (e.g., model M196) often outperform those trained on the entire aggregate (M246). This suggests that certain datasets introduce noise or highly specific patterns that hinder generalization.
Figure 2 illustrates the cross-domain performance across various training and test sets. The heatmap reveals significant asymmetry; for instance, a model trained on botwiki-verified performs well on vendor-verified, but the reverse is less effective. This highlights the importance of "separability" in training data.
The optimized model (M196) achieved an AUC of 0.99 on holdout sets like botwiki and midterm-18. Its efficiency allows for the evaluation of 900 million tweets per day on a single CPU, a 200x speedup over tools like Botometer.
Discussion and Limitations
While highly effective for individual account detection, the model struggles with coordinated campaigns where accounts appear human in isolation but act in sync (e.g., cresci-stock). Future work may involve instance-level data selection to further refine training sets and mitigate adversarial evasion by sophisticated bot operators who "groom" profile metadata to appear more human.
Summary updated