Statistical Detection of Online Drifting Twitter Spam: Invited Paper

Shigang Liu, Jun Zhang, Yang XiangAsiaCCS 2016 · DOI

TL;DR

This paper tackles the challenge of "Twitter spam drift," where spam characteristics change over time, rendering traditional detection models obsolete. The authors propose a novel method combining fuzzy-based redistribution and asymmetric sampling to create more robust spam classifiers that adapt to these evolving patterns, significantly improving detection performance.

What Problem Does This Paper Address?

The proliferation of spam on social media platforms like Twitter poses a significant security and user experience problem. While machine learning has been applied to detect Twitter spam, a critical issue known as "Twitter spam drift" undermines these efforts. This drift means that the statistical features of spam tweets evolve over time, causing classifiers trained on older data to become ineffective. A common solution is to retrain classifiers daily, but this is hampered by the scarcity and imbalance of labeled spam data, where spam instances are vastly outnumbered by legitimate tweets. This imbalance can lead classifiers to be biased towards the majority class, performing poorly on the minority spam class. This paper aims to address both the spam drift and the imbalanced data challenges in Twitter spam detection.

Methods

The proposed method introduces two core techniques to combat spam drift and imbalanced training data:

  • Fuzzy-Based Redistribution: This technique aims to generate more synthetic spam samples to alleviate the class imbalance. It uses an information decomposition approach to create reliable synthetic spam data. The process involves partitioning feature value ranges into small intervals, calculating weights based on observed data, and then generating synthetic values for the spam class. This method is designed to maintain the independence of features in the generated samples.
  • Asymmetric Sampling: This technique re-balances the training data by applying different sampling strategies to the spam (minority) and non-spam (majority) classes. Specifically, it uses an over-sampling strategy for spam samples (e.g., by duplicating them) and an under-sampling strategy for non-spam samples (by randomly removing them). This aims to create a more balanced dataset for training.

These techniques are integrated into an ensemble framework. The method trains classifiers on two different training sets: one using the original imbalanced data and another using data augmented by fuzzy-based redistribution and asymmetric sampling. The final decision is made by combining the predictions from these ensemble classifiers, often using a majority voting rule.

Datasets

The authors used a real-world 10-day ground-truth dataset collected using Twitter's Streaming API. This dataset was labeled with the help of internal tools from Trend Micro. The dataset consists of over 600 million tweets. For experiments, they used a subset of 1,000 spam tweets and 10,000 non-spam tweets for training, and various configurations for testing, simulating different spam-to-non-spam ratios (10:1 and 100:1).

Key Insights and Ideas

  • Spam Drift is a Real and Significant Problem: The paper provides compelling visual evidence (Figures 1 and 2) showing how statistical features of spam tweets (character count, account age) change significantly over just 10 days, directly impacting classifier performance.
  • Daily Retraining is Necessary but Insufficient: While retraining daily is a logical response to spam drift, it exacerbates the problem of imbalanced data, leading to biased classifiers.
  • Fuzzy Logic for Synthetic Data Generation: The use of fuzzy logic for generating synthetic spam samples is an innovative way to increase the representation of the minority class without relying solely on existing, scarce samples.
  • Asymmetric Sampling is Key to Balancing: The paper highlights that a balanced approach to sampling (over-sampling minority, under-sampling majority) is crucial for effective classifier training on imbalanced datasets.
  • Ensemble Methods Enhance Robustness: Combining classifiers trained on different data distributions (original vs. augmented) through an ensemble approach significantly boosts overall detection performance and robustness against drift.

Key Results

The proposed method demonstrated significant improvements in detecting drifting Twitter spam compared to traditional methods like Random Forest (RF) and C4.5 Decision Trees.

  • Overall Performance: The new method consistently outperformed C4.5 and RF across various metrics. For instance, in terms of detection rate, the new method was up to 10% better than C4.5 and significantly better than RF. For AUC, the new method was over 5% higher than the other methods.
  • Impact of Spam Drift: Figures 4, 6, 7, 8, and 9 clearly show that classifiers trained on older data (e.g., "day1 data") degrade significantly over time, while classifiers trained daily ("real-time data") perform much better. The proposed method aims to bridge this gap.
  • Day-Based Performance: The proposed method showed superior performance in both high and low spam rate testing scenarios. For example, in testing set-1 (10% spam rate), the detection rate of the new method was up to 20% higher than RF. In testing set-2 (1% spam rate), the new method's AUC was up to 10% higher than RF and C4.5.
  • Statistical Significance: ANOVA and Tukey's HSD tests confirmed that the new detection method achieved statistically significant improvements over C4.5 and RF in most cases.

Limitations and Future Work

  • Limitations: The paper acknowledges the challenge of obtaining a large number of labeled spam samples, which is a common issue in spam detection research. The reliance on lightweight features, while necessary for real-time detection, might limit the discriminative power compared to more complex social graph features.
  • Future Work: The paper suggests that the proposed method can effectively address the problem of small, imbalanced training data and improve detection performance for drifting Twitter spam. Further research could explore more advanced feature extraction techniques or investigate the application of this method to other evolving spam detection scenarios.

Summary updated

BibTeX
@inproceedings{liu2016statistical,
  author = {Shigang Liu and Jun Zhang and Yang Xiang},
  title = {{Statistical Detection of Online Drifting Twitter Spam: Invited Paper}},
  booktitle = {AsiaCCS},
  year = {2016},
  doi = {10.1145/2897845.2897928},
}
Figure