Identifying Coordinated Accounts on Social Media through Hidden Influence and Group Behaviours

Karishma Sharma, Yizhou Zhang, Emilio Ferrara, Yan Liu

TL;DR

Researchers have developed AMDN-HAGE, an unsupervised generative model that identifies coordinated "troll" accounts by analyzing their latent influence on one another and their collective anomalous behavior. Unlike previous methods, it doesn't require labeled data or rigid assumptions like perfect timing synchronization, successfully uncovering disinformation campaigns related to the 2016 U.S. Election and the COVID-19 pandemic.

What Problem Does This Paper Address?

Social media is frequently weaponized by coordinated groups—often called "troll farms"—to manipulate public opinion through disinformation. Detecting these groups is a cat-and-mouse game. Existing detection methods typically fall into two traps:

  1. Rigid Heuristics: They assume coordinated accounts act in perfect lockstep (e.g., tweeting at the exact same second). Sophisticated human-operated campaigns easily bypass these filters by varying their timing.
  2. Supervision Dependency: They require a pre-labeled set of "bad" accounts to train a classifier, making them useless for discovering entirely new, unknown campaigns.

This paper addresses the need for a flexible, unsupervised approach that can identify coordination based on the intrinsic characteristics of how these accounts interact and influence each other over time.

Methods

The authors propose a framework called AMDN-HAGE (Attentive Mixture Density Network with Hidden Account Group Estimation). The model consists of two main components that share a joint embedding space:

  • AMDN (Activity Trace Modeling): This part uses Temporal Point Processes (TPP) enhanced with masked self-attention. It treats every tweet or retweet as an "event" and learns how past events from any account influence the probability of a future event from a specific account. This captures the "hidden influence" that coordinated accounts exert on each other.
  • HAGE (Social Group Modeling): This component uses a Gaussian Mixture Model (GMM) to cluster accounts in the embedding space. Coordinated groups are identified as "anomalous" clusters—groups that are either very small or have unusually low variance compared to the general population.
Figure 2: Architecture of proposed (AMDN-HAGE) to model conditional density function of account activities and hid- den groups on social media.

To train this complex system, the authors developed a bilevel optimization algorithm. It alternates between Expectation-Maximization (EM) for the GMM parameters and Stochastic Gradient Descent (SGD) for the neural network. This was necessary because standard optimizers like Adam often fail to respect the mathematical constraints of GMMs, leading to numerical instability.

Datasets

The model was evaluated on two significant real-world datasets:

  1. Russian IRA Dataset: A set of 2,025 accounts (312 confirmed Russian "trolls" and 1,713 normal users) active during the 2016 U.S. Presidential Election. This provided a ground-truth benchmark.
  2. COVID-19 Dataset: A massive, newly collected dataset of 119,298 active accounts and 13.9 million tweets from March 2020 to July 2021. This served as a "wild" test case to find previously unknown coordination.

Key Insights and Ideas

  • Coordination is Temporal, Not Just Synchronous: The authors found that coordinated activity isn't always perfectly timed. As shown in Figure 1, the delay between coordinated tweets can range from a few hours to several days.
Figure 1: Coordinated accounts suspended by Twitter in COVID-19 data (also identified with proposed method). Ex- ample tweets from these accounts show political conspira- cies, promoted in coordination. The time difference of their coordinated activity varies from less than 6 hours in one in- stance to more than half a week in the other.
  • High Influence, Fast Decay: Coordinated accounts show significantly higher mutual influence than normal account pairs. However, this influence decays much faster over time. Coordination is often an intense, short-lived burst of activity designed to amplify a specific narrative quickly.
  • Optimization Matters: Direct optimization of the joint likelihood function using standard tools like Adam proved unstable, often resulting in "NaN" (Not a Number) errors. The proposed bilevel approach was critical for convergence.
Figure 3: Comparison of bilevel optimization and Adam.

Key Results

  • Superior Detection: On the IRA dataset, AMDN-HAGE achieved an Average Precision (AP) of 0.804 in a completely unsupervised setting. This drastically outperformed traditional baselines like Co-activity clustering (0.208) and Hawkes Processes (0.337).
  • Real-World Discovery: In the COVID-19 data, the model identified two anomalous clusters. Manual inspection revealed these groups were spreading anti-vaccine and anti-mask conspiracies (using hashtags like #Plandemic and #CovidHoax) and were heavily linked to the QAnon movement.
  • Account Age: A statistical analysis of the COVID-19 clusters showed that the anomalous groups had a much higher proportion of accounts created very recently (2019-2020), a common hallmark of "disposable" accounts used in influence operations.

Limitations and Future Work

While highly effective, the authors note a few areas for growth:

  • Feature Integration: The current model relies almost entirely on account IDs and timestamps. While this makes it language-independent, incorporating text content or account metadata (like profile descriptions) could further improve accuracy.
  • Platform Specificity: The study focused on Twitter. Future work could explore how these coordination patterns manifest across different platforms (e.g., cross-posting between Reddit and Twitter).
  • False Positives: Small, highly active legitimate groups (like a tight-knit community of hobbyists) might occasionally be flagged as anomalous, requiring further refinement to distinguish "concerted" legitimate activity from "coordinated" malicious activity.

Summary updated

Abstract

Disinformation campaigns on social media, involving coordinated activities from malicious accounts towards manipulating public opinion, have become increasingly prevalent. Existing approaches to detect coordinated accounts either make very strict assumptions about coordinated behaviours, or require part of the malicious accounts in the coordinated group to be revealed in order to detect the rest. To address these drawbacks, we propose a generative model, AMDN-HAGE (Attentive Mixture Density Network with Hidden Account Group Estimation) which jointly models account activities and hidden group behaviours based on Temporal Point Processes (TPP) and Gaussian Mixture Model (GMM), to capture inherent characteristics of coordination which is, accounts that coordinate must strongly influence each other's activities, and collectively appear anomalous from normal accounts. To address the challenges of optimizing the proposed model, we provide a bilevel optimization algorithm with theoretical guarantee on convergence. We verified the effectiveness of the proposed method and training algorithm on real-world social network data collected from Twitter related to coordinated campaigns from Russia's Internet Research Agency targeting the 2016 U.S. Presidential Elections, and to identify coordinated campaigns related to the COVID-19 pandemic. Leveraging the learned model, we find that the average influence between coordinated account pairs is the highest.On COVID-19, we found coordinated group spreading anti-vaccination, anti-masks conspiracies that suggest the pandemic is a hoax and political scam.
BibTeX
@inproceedings{sharma2021identifying,
  author = {Karishma Sharma and Yizhou Zhang and Emilio Ferrara and Yan Liu},
  title = {{Identifying Coordinated Accounts on Social Media through Hidden Influence and Group Behaviours}},
  booktitle = {ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year = {2021},
  url = {https://openreview.net/forum?id=ZAA-np8yJPu},
}
Figure