Abstract
As large language models (LLMs) scale, concerns about their potential for automated influence operations have intensified. This study empirically evaluates the relationship between model size and political persuasiveness using a randomized survey experiment with 25,982 participants and 24 different models. We identify a log scaling law: persuasive impact increases linearly with the logarithm of parameter count. Mid-sized models (e.g., 7B parameters) are found to be nearly as persuasive as frontier models like GPT-4, primarily because persuasion in this context is driven by basic "task completion"—the ability to generate coherent, on-topic arguments.
Introduction
The rapid advancement of LLMs has sparked significant AI safety concerns regarding their potential to automate high-stakes influence operations, propaganda, and disinformation. While benchmarks often focus on static capabilities like mathematics or coding, persuasion is a sociotechnical task that requires measuring human responses. This study addresses the gap in empirical understanding of how model scaling affects persuasive impact and whether a functional ceiling exists.
Methodology
The researchers conducted a massive randomized survey experiment to estimate the persuasive capability of 24 language models, ranging from 70M to over 300B parameters.
- Model Selection: 22 open-weight models from five families (Pythia, Qwen1.5, Llama-2, Yi, and Falcon) and two frontier models (GPT-4-Turbo and Claude-3-Opus).
- Standardization: All open-weight base models were instruction-tuned on the GPT-4 Alpaca dataset (10,000 examples) using LoRA to ensure consistent instruction-following capabilities.
- Experimental Design: Participants were randomly assigned to a control group, a human-written message group, or an AI-generated message group across 10 contemporary U.S. political issues (e.g., carbon taxes, Medicaid work requirements).
- Analysis: A random-effects meta-analysis modeled the relationship between the natural logarithm of parameter count and the Average Treatment Effect (ATE) on participant attitudes.
The Log Scaling Law
The study's primary finding is that persuasiveness increases linearly with the logarithm of the parameter count. This implies that exponential increases in model size yield only linear gains in persuasive power.
As shown in Figure 2, while larger models are generally more effective, they hit a ceiling of diminishing returns. Frontier models were not significantly more persuasive than mid-sized models like Qwen1.5-7B or Llama-2-13B. In many cases, these 7B-class models already match human-level persuasiveness.
Persuasion as Task Completion
To understand why scaling benefits plateau, the authors analyzed message features such as length, sentiment, and readability. They found that "task completion"—the ability to stay on topic and argue for the assigned stance—was the strongest predictor of persuasive success.
Figure 5 illustrates that once a model reaches a certain threshold of coherence (typically around the 7B parameter mark), further scaling provides minimal additional benefit for static, non-targeted messages. The "bottleneck" is not model intelligence, but the inherent difficulty of changing a human's mind with a single paragraph of text.
Conclusion
The existence of a log scaling law suggests that the threat of "superhuman" persuasion from scaling alone may be overstated for static messages. However, the authors caution that these laws might differ for multi-turn dialogues or personalized microtargeting. Future research should explore whether "persuasion-tuned" models or dynamic interactions break the observed log scaling relationship.
Summary updated