Simulating Opinion Dynamics with Networks of LLM-based Agents

Yun-Shiuan Chuang, Agam Goyal, Nikunj Harlalka, Siddharth Suresh, Robert Hawkins, Sijia Yang, Dhavan Shah, Junjie Hu, Timothy T. Rogers

Abstract

Accurately simulating human opinion dynamics is crucial for understanding a variety of societal phenomena, including polarization and the spread of misinformation. However, the agent-based models (ABMs) commonly used for such simulations lack fidelity to human behavior. We propose a new approach to simulating opinion dynamics based on populations of Large Language Models (LLMs). Our findings reveal a strong inherent bias in LLM agents towards accurate information, leading to consensus in line with scientific reality. However, this bias limits the simulation of individuals with resistant views on issues like climate change. After inducing confirmation bias through prompt engineering, we observed opinion fragmentation in line with existing agent-based research. These insights highlight the promise and limitations of LLM agents in this domain and suggest a path forward: refining LLMs with real-world discourse to better simulate the evolution of human beliefs.

Introduction

For decades, researchers in sociology and economics have relied on Agent-Based Models (ABMs) to study the spread of ideas and the formation of consensus. Traditionally, these models represent agents as simplified entities where opinions are numerical scalars (e.g., a value between -1 and 1). While mathematically tractable, this approach fails to capture the linguistic nuances, demographic backgrounds, and complex reasoning that define human social interaction.

The emergence of Large Language Models (LLMs) presents an opportunity to bridge this gap. Unlike traditional ABMs, LLM-based agents can process context, adopt specific personas, and engage in natural language dialogue. This paper investigates whether networks of LLMs can realistically simulate how opinions evolve, polarize, or reach consensus within a social group.

Comparison between LLM Generative Agents and Classic Agent-Based Models.

Methodology

The researchers developed a framework where a population of $N=10$ LLM agents (utilizing gpt-3.5-turbo-16k) interact over a series of $T=100$ steps. Each agent is assigned a distinct demographic persona, including age, ethnicity, occupation, and political leaning.

The Interaction Loop and Memory

In each simulation step, a "speaker" agent is selected to write a tweet reflecting its current beliefs. A "listener" agent then reads the tweet, reports an internal reaction, and updates its internal state. To manage this state, the researchers implemented a sophisticated memory module. They compared two architectures: Cumulative memory, which stores every interaction, and Reflective memory, which periodically summarizes experiences to maintain a compact and manageable state.

Overview of the LLM-based agent network (a) and the internal memory architecture (b).

Experimental Design

The study covered 15 topics across Science, History, and Common Sense. To analyze the agents' verbal reports, a separate model (FLAN-T5-XXL) acted as a classifier, mapping natural language responses onto a 5-point Likert scale. The experiments manipulated two key variables:

  1. Framing: Presenting a topic as either a factual truth or a conspiracy theory.
  2. Confirmation Bias: Instructing agents to either be open-minded or to exhibit "confirmation bias" by dismissing contradictory information.

Key Insights

The Inherent Truth Bias

A significant finding was the "truth bias" inherent in LLMs. Without explicit instructions to be biased, agents consistently converged toward scientific reality, even when the simulation was framed around misinformation. The agents effectively "debugged" false claims through social interaction, suggesting that current LLMs are heavily aligned toward factual accuracy due to their training and Reinforcement Learning from Human Feedback (RLHF).

Replicating Polarization

To achieve realistic opinion fragmentation and polarization—common features of human discourse—the researchers had to explicitly prompt for confirmation bias. This indicates that the "stubbornness" often seen in human society is not an emergent property of the LLM's basic language processing but must be intentionally injected. When confirmation bias was induced, the diversity of opinions ($D$) rose significantly (from ~0.6 to ~1.6), and the population split into echo chambers.

Results and Discussion

Under "False Framing" scenarios, agents without induced bias showed a strong negative bias ($B = -1.33$), successfully rejecting misinformation. However, the study also found that even if a population began with a false belief, they would eventually shift toward the truth through interaction, provided they lacked strong confirmation bias.

The findings remained consistent across different models, including GPT-4 and Vicuna-33B, suggesting these behaviors are fundamental to current LLM architectures. Furthermore, the researchers addressed potential "hallucinations" (agents citing non-existent external evidence) by implementing a "closed-world" prompt, which restricted agents to the information provided within the simulation.

Limitations and Future Directions

The primary limitation identified is the influence of RLHF, which makes it difficult to simulate populations genuinely susceptible to fringe theories without heavy prompting. Additionally, the current study utilized a "complete graph" network topology where every agent can interact with every other agent. Real-world social networks typically exhibit "small-world" properties and homophily, which were not explored here.

Future research should focus on fine-tuning LLMs on actual human discourse data from platforms like Reddit or Twitter. This could help capture more authentic, less "polite" social behaviors and provide a more accurate reflection of human belief evolution.

Summary updated

Abstract

Accurately simulating human opinion dynamics is crucial for understanding a variety of societal phenomena, including polarization and the spread of misinformation. However, the agent-based models (ABMs) commonly used for such simulations lack fidelity to human behavior. We propose a new approach to simulating opinion dynamics based on populations of Large Language Models (LLMs). Our findings reveal a strong inherent bias in LLM agents towards accurate information, leading to consensus in line with scientific reality. However, this bias limits the simulation of individuals with resistant views on issues like climate change. After inducing confirmation bias through prompt engineering, we observed opinion fragmentation in line with existing agent-based research. These insights highlight the promise and limitations of LLM agents in this domain and suggest a path forward: refining LLMs with real-world discourse to better simulate the evolution of human beliefs.
BibTeX
@inproceedings{chuang2024simulating,
  author = {Yun-Shiuan Chuang and Agam Goyal and Nikunj Harlalka and Siddharth Suresh and Robert Hawkins and Sijia Yang and Dhavan Shah and Junjie Hu and Timothy T. Rogers},
  title = {{Simulating Opinion Dynamics with Networks of LLM-based Agents}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=XRVCjYUOhF},
}
Figure