Unraveling the Web of Disinformation: Exploring the Larger Context of State-Sponsored Influence Campaigns on Twitter

Mohammad Hammas Saeed, Shiza Ali, Pujan Paudel, Jeremy Blackburn, Gianluca Stringhini

Abstract

Social media platforms offer unprecedented opportunities for connectivity and exchange of ideas; however, they also serve as fertile grounds for the dissemination of disinformation. Over the years, there has been a rise in state-sponsored campaigns aiming to spread disinformation and sway public opinion on sensitive topics through designated accounts, known as troll accounts. Past works on detecting accounts belonging to state-backed operations focus on a single campaign. While campaign-specific detection techniques are easier to build, there is no work done on developing systems that are campaign-agnostic and offer generalized detection of troll accounts unaffected by the biases of the specific campaign they belong to. In this paper, we identify several strategies adopted across different state actors and present a system that leverages them to detect accounts from previously unseen campaigns. We study 19 state-sponsored disinformation campaigns that took place on Twitter, originating from various countries. The strategies include sending automated messages through popular scheduling services, retweeting and sharing selective content and using fake versions of verified applications for pushing content. By translating these traits into a feature set, we build a machine learning-based classifier that can correctly identify up to 94% of accounts from unseen campaigns. Additionally, we run our system in the wild and find more accounts that could potentially belong to state-backed operations. We also present case studies to highlight the similarity between the accounts found by our system and those identified by Twitter.

Introduction

Detecting state-sponsored influence operations on social media is a persistent challenge. Historically, detection systems have been campaign-specific, trained to recognize keywords or behaviors tied to a single event, such as the 2016 US Election. However, these models are often brittle, failing when new actors emerge or tactics shift. State-sponsored "trolls" are increasingly sophisticated, exhibiting human-like behaviors and blending into communities to avoid detection. This research proposes a campaign-agnostic system designed to identify the "universal fingerprints" of state-backed operations regardless of their origin or target.

Methodology: Identifying Universal Fingerprints

The study analyzed 19 state-sponsored campaigns involving nearly 80,000 accounts and 200 million tweets. To build a generalized classifier, the researchers developed a set of 45 features across four categories:

  1. User Attributes: Metadata such as account age, follower counts, and tweet frequency.
  2. Temporal Characteristics: 24-hour activity fingerprints.
  3. Stylometry: Linguistic markers like sentence complexity and the ratio of function words.
  4. Source Features: Tracking the specific applications used to post (e.g., "Twitter for iPhone" vs. custom scheduling tools).
Figure 5: Cumulative Distribution Function (CDF) of the number of sources used by real users versus state-sponsored accounts, highlighting the distinct source-usage patterns of troll operations.

The researchers employed a "leave-one-campaign-out" training approach to ensure the model could detect accounts from campaigns it had never encountered during training.

Key Behavioral Insights

The analysis revealed several cross-campaign strategies:

  • The "Fake App" Trick: State actors frequently use impersonated third-party applications with names that look legitimate but contain subtle errors (e.g., "Twitter for Android " with an extra space).
  • Camouflage via "Advice": Trolls often post verbatim "advice" messages—inspirational quotes or uplifting content—shared across entirely different campaigns to build social proof and appear human.
  • Shared Infrastructure: Synchronized spikes in the use of specific custom applications across different state actors suggest shared tactics or service providers.
  • Amplification Tactics: High "Retweet Fractions" were strong predictors of troll accounts, as these operations rely heavily on amplification to force narratives into the public eye.
Figure 3: Comparison of (a) the percentage of messages from scheduling apps and (b) the percentage of messages that are retweets across 19 different state-sponsored campaigns.

Results and Real-World Impact

The Random Forest classifier achieved a standard cross-validation accuracy of 98.5%. More importantly, when trained on a single campaign (e.g., the 2018 Russian IRA), it could detect up to 94% of accounts in unrelated, unseen campaigns.

In a real-world test, the system analyzed 2,696 accounts using "fake" applications and identified 116 new malicious accounts not yet officially flagged by Twitter. The system maintained a low false-positive rate of approximately 1.75%.

Conclusion and Future Directions

While the system shows great promise, the evolving landscape of social media presents new hurdles. The discontinuation of free API access limits large-scale real-time testing, and the rise of Large Language Models (LLMs) may alter the stylometric fingerprints of future troll campaigns. Future research must adapt to these generative AI threats and the potential shift toward more decentralized, manual posting methods by state actors.

Summary updated

Abstract

Social media platforms offer unprecedented opportunities for connectivity and exchange of ideas; however, they also serve as fertile grounds for the dissemination of disinformation. Over the years, there has been a rise in state-sponsored campaigns aiming to spread disinformation and sway public opinion on sensitive topics through designated accounts, known as troll accounts. Past works on detecting accounts belonging to state-backed operations focus on a single campaign. While campaign-specific detection techniques are easier to build, there is no work done on developing systems that are campaign-agnostic and offer generalized detection of troll accounts unaffected by the biases of the specific campaign they belong to. In this paper, we identify several strategies adopted across different state actors and present a system that leverages them to detect accounts from previously unseen campaigns. We study 19 state-sponsored disinformation campaigns that took place on Twitter, originating from various countries. The strategies include sending automated messages through popular scheduling services, retweeting and sharing selective content and using fake versions of verified applications for pushing content. By translating these traits into a feature set, we build a machine learning-based classifier that can correctly identify up to 94% of accounts from unseen campaigns. Additionally, we run our system in the wild and find more accounts that could potentially belong to state-backed operations. We also present case studies to highlight the similarity between the accounts found by our system and those identified by Twitter.
BibTeX
@inproceedings{saeed2024unraveling,
  author = {Mohammad Hammas Saeed and Shiza Ali and Pujan Paudel and Jeremy Blackburn and Gianluca Stringhini},
  title = {{Unraveling the Web of Disinformation: Exploring the Larger Context of State-Sponsored Influence Campaigns on Twitter}},
  booktitle = {RAID},
  year = {2024},
  url = {https://openreview.net/forum?id=B29PB8Gqs9},
}
Figure