Unraveling the Web of Disinformation: Exploring the Larger Context of State-Sponsored Influence Campaigns on Twitter

Mohammad Hammas Saeed, Shiza Ali, Pujan Paudel, Jeremy Blackburn, Gianluca Stringhini

TL;DR

Researchers have developed a machine learning system capable of detecting state-sponsored "troll" accounts by identifying universal behavioral traits that transcend specific countries or political events. By focusing on shared infrastructure and tactics—such as the use of impersonated third-party apps and "advice" messages—their model can identify up to 94% of accounts from campaigns it has never seen before.

What Problem Does This Paper Address?

Detecting state-sponsored influence operations on social media is a constant cat-and-mouse game. Historically, detection systems have been campaign-specific, meaning they are trained to recognize the specific keywords or behaviors of a single event (like the 2016 US Election or the 2018 Brazilian elections).

The problem is that these models are brittle; they often fail when a new state actor enters the fray or when tactics shift. Furthermore, state-sponsored trolls are more sophisticated than simple "spam bots"—they often exhibit human-like behavior, engage in "copypasta" networks, and blend into communities to avoid detection. The authors sought to build a campaign-agnostic system: one that identifies the "universal fingerprints" of state-backed operations regardless of their origin or target.

Methods

The researchers analyzed 19 different state-sponsored campaigns (spanning countries like Russia, China, Iran, and Venezuela) involving nearly 80,000 accounts and 200 million tweets.

They translated their observations into a set of 45 features categorized into four groups:

  1. User Attributes: Standard metadata like account age, follower counts, and tweet frequency.
  2. Temporal Characteristics: A 24-hour "activity fingerprint" showing when the account typically posts.
  3. Stylometry: Linguistic markers including sentence complexity, word length, and the ratio of "function words" (like articles and prepositions) to "non-function words."
  4. Source Features: This is a key novelty. They tracked the specific applications used to post (e.g., "Twitter for iPhone" vs. a scheduling app like "Hootsuite").

They trained several classifiers, with the Random Forest model performing best. To test "unseen" detection, they used a "leave-one-campaign-out" approach: training the model on one known campaign and testing its ability to flag accounts in the others.

Datasets

  • Twitter Transparency Dataset: A collection of data released by Twitter (now X) regarding accounts officially attributed to state-backed operations between 2018 and 2022.
  • Twitter 1% Data: A random 1% sample of public tweets used as a "benign" baseline to represent regular, non-troll users.
  • Twitter Search API: Used to supplement the dataset with historical tweets for both malicious and benign accounts.

Key Insights and Ideas

  • The "Fake App" Trick: One of the most striking findings was the use of impersonated third-party applications. State actors often use custom tools with names that look legitimate at a glance but contain subtle errors, such as "Twitter for Android " (with an extra space) or "Twitter for iphons". These apps are not available on official stores and are used almost exclusively by coordinated campaigns.
  • Camouflage via "Advice": Trolls don't just post propaganda. To build "social proof" and appear human, they frequently post verbatim "advice" messages—inspirational quotes or uplifting content—that are shared across entirely different campaigns (e.g., a Russian campaign and a Saudi campaign using the same "uplifting" text).
  • Shared Infrastructure: The researchers found synchronized spikes in the use of specific custom applications (like "Moments Internal Auth") across different state actors. This suggests that these groups might be sharing tactics, using the same "troll farm" service providers, or operating under a shared strategic alliance.
  • Predictive Power of Retweets: The "Retweet Fraction" and "Cumulative Mentions" were the strongest predictors of a troll account, confirming that these operations rely heavily on amplification and tagging to force their narratives into the public eye.

Key Results

  • High Accuracy: In standard 10-fold cross-validation, the Random Forest model achieved 98.5% accuracy and an F1-score of 97.8%.
  • Cross-Campaign Success: When trained on the 2018 Russian IRA campaign, the system could detect 94% of accounts in other, unrelated campaigns. Even the weakest training sets (campaigns with very few accounts) still allowed the model to flag a significant portion of unseen trolls.
  • In-the-Wild Detection: The system was run on a fresh set of 2,696 accounts using "fake" applications and successfully flagged 116 new malicious accounts that had not yet been officially identified by Twitter.
  • Low False Positives: The system maintained a very low false-positive rate, estimated at roughly 1.75% in the worst-case scenario.

Limitations and Future Work

  • The API Hurdle: The discontinuation of the free Twitter API during the study significantly hampered the researchers' ability to test the system on larger, real-time datasets.
  • The LLM Era: The dataset used in this study predates the widespread availability of Large Language Models (LLMs) like ChatGPT. Future work will need to address how generative AI might change the "stylometric fingerprint" of troll accounts, potentially making them harder to distinguish from humans.
  • Label Noise: The "benign" dataset (the 1% stream) might inadvertently contain trolls that haven't been caught yet, which could introduce slight biases into the training process.
  • Evasion Tactics: As detection becomes more holistic, state actors may move away from custom apps and scheduling services to more manual, decentralized posting methods to blend in further.

Summary updated

Abstract

Social media platforms offer unprecedented opportunities for connectivity and exchange of ideas; however, they also serve as fertile grounds for the dissemination of disinformation. Over the years, there has been a rise in state-sponsored campaigns aiming to spread disinformation and sway public opinion on sensitive topics through designated accounts, known as troll accounts. Past works on detecting accounts belonging to state-backed operations focus on a single campaign. While campaign-specific detection techniques are easier to build, there is no work done on developing systems that are campaign-agnostic and offer generalized detection of troll accounts unaffected by the biases of the specific campaign they belong to. In this paper, we identify several strategies adopted across different state actors and present a system that leverages them to detect accounts from previously unseen campaigns. We study 19 state-sponsored disinformation campaigns that took place on Twitter, originating from various countries. The strategies include sending automated messages through popular scheduling services, retweeting and sharing selective content and using fake versions of verified applications for pushing content. By translating these traits into a feature set, we build a machine learning-based classifier that can correctly identify up to 94% of accounts from unseen campaigns. Additionally, we run our system in the wild and find more accounts that could potentially belong to state-backed operations. We also present case studies to highlight the similarity between the accounts found by our system and those identified by Twitter.
BibTeX
@inproceedings{saeed2024unraveling,
  author = {Mohammad Hammas Saeed and Shiza Ali and Pujan Paudel and Jeremy Blackburn and Gianluca Stringhini},
  title = {{Unraveling the Web of Disinformation: Exploring the Larger Context of State-Sponsored Influence Campaigns on Twitter}},
  booktitle = {RAID},
  year = {2024},
  url = {https://openreview.net/forum?id=B29PB8Gqs9},
}

Loading related content...

Figure