Abstract
Adversarial information operations can destabilize societies by undermining fair elections, manipulating public opinions on policies, and promoting scams. Despite their widespread occurrence and potential impacts, our understanding of influence campaigns is limited by manual analysis of messages and subjective interpretation of their observable behavior. In this paper, we explore whether these limitations can be mitigated with large language models (LLMs), using GPT-3.5 as a case-study for coordinated campaign annotation. We first use GPT-3.5 to scrutinize 126 identified information operations spanning over a decade. We utilize a number of metrics to quantify the close (if imperfect) agreement between LLM and ground truth descriptions. We next extract coordinated campaigns from two large multilingual datasets from X (formerly Twitter) that respectively discuss the 2022 French election and 2023 Balikaran Philippine-U.S. military exercise in 2023. For each coordinated campaign, we use GPT-3.5 to analyze posts related to a specific concern and extract goals, tactics, and narrative frames, both before and after critical events. While the GPT-3.5 sometimes disagrees with subjective interpretation, its ability to summarize and interpret demonstrates LLMs' potential to extract higher-order indicators from text to provide a more complete picture of the information campaigns compared to previous methods.
Introduction
Adversarial information operations (info-ops) are a constant threat to democratic processes and social stability. While researchers have become proficient at detecting coordinated behavior—such as identifying botnets or groups of accounts posting identical content—understanding the "higher-order" intent behind these campaigns remains a significant bottleneck. Traditionally, determining a campaign's political goals or rhetorical tactics required subject matter experts to manually read and annotate thousands of messages. This process is subjective, expensive, and impossible to scale to the volume of modern social media. This research explores whether LLMs can bridge this gap by automatically categorizing the concerns, tactics, and framing of influence operations.
Methodology
The researchers developed a multi-stage pipeline designed to move from raw social media data to high-level narrative analysis.
The pipeline consists of four primary stages:
- Coordination Detection: Utilizing a validated hashtag co-occurrence heuristic to link accounts that post identical strings of hashtags in the same order.
- Concern Labeling (Knowledge Distillation): To categorize millions of posts into specific "wedge issues," the team used a cost-effective distillation approach. GPT-4 labeled a small subset of data, which was then used to instruction-tune a smaller, open-source Llama-2-7B model for large-scale classification.
- Temporal Extraction: Splitting the data based on critical events, such as election dates or military exercise start times.
- LLM Data Analysis: Applying GPT-3.5 to coordinated clusters to extract features across established social science frameworks: identifying targeted countries and political goals, categorizing tactics (BEND Framework), and identifying narrative frames (Framing Theory).
Datasets and Case Studies
The study utilized three primary data sources to validate and test the pipeline:
- Benchmark Dataset: 126 previously identified and manually annotated info-ops used to validate LLM accuracy.
- 2022 French Election Dataset: 5.9 million posts collected around the presidential election.
- 2023 Balikatan Dataset: 4.7 million posts regarding U.S.-Philippines military exercises.
In the Balikatan dataset, the LLM demonstrated its utility in filtering noise. It quickly identified that the largest "coordinated" cluster was not a political influence operation, but a commercial spam campaign promoting a Bollywood movie. In the French election data, the researchers observed narrative evolution: the "remedy" suggested by coordinated accounts shifted from "stopping Russian aggression" before the election to "negotiating safe passage for civilians" as the conflict progressed.
Results
The validation against 126 known campaigns showed strong agreement between GPT-3.5 and ground truth for "Targeted Country" and "Event Description." However, the model faced challenges with "Political Goal Category" in zero-shot settings.
In the French election dataset, coordinated accounts were responsible for 45.5% of all posts, despite representing only 11.6% of authors. While the LLM was highly accurate at identifying the Explain tactic (providing details), it occasionally struggled with sentiment-heavy tactics, sometimes mistaking urgent calls to action for "excitement" regardless of sentiment.
Discussion and Future Work
While LLMs offer a scalable way to analyze complex influence campaigns, limitations remain. Hallucinations still occur, necessitating "human-in-the-loop" verification for high-stakes findings. Additionally, the concern detection model performed better on frequent topics than on rarer issues like "Public Services." Future research will explore more powerful models like Llama-3 and incorporate "Chain-of-Thought" prompting to improve reasoning regarding complex political goals.
Summary updated