Social science has long been constrained by the "human bottleneck": the high cost and ethical complexity of recruiting thousands of participants for experiments. The rise of Large Language Models (LLMs) offers a tantalizing alternative. If a model can mimic human reasoning, could we use it as a "silicon participant" to pilot surveys, test economic theories, or simulate entire societies?
While researchers are increasingly building "social sandboxes" like the one proposed by Maximilian Puelma Touzel and colleagues in A Simulation System Towards Solving Societal-Scale Manipulation, a critical question remains: are these models actually behaving like humans, or are they just sophisticated statistical mimics?
The Mirage of Survey Alignment
The most common way to "measure" an LLM’s humanity is to give it a multiple-choice survey. If the model answers like a median voter, we assume it "understands" that demographic. However, Ricardo Dominguez-Olmedo, Moritz Hardt, and Celestine Mendler-Dünner argue in Questioning the Survey Responses of Large Language Models that these results are often a mirage.
They found that LLM responses are heavily influenced by "labeling biases"—such as a simple preference for choosing the letter 'A' regardless of the content. Once these structural artifacts are removed, the models often revert to a uniform random distribution rather than reflecting a specific human population. This suggests that models aren't "holding" opinions; they are navigating the constraints of their prompt.
The Utopian Illusion and Social Desirability
Even when models do exhibit human-like patterns, they often suffer from what Ning Bian and Xianpei Han call the Utopian Illusion. In Social Simulations with Large Language Model Risk Utopian Illusion, they show that LLM-driven societies tend to be overly polite, cooperative, and idealized.
This is largely due to social desirability bias. As Aadesh Salecha and Lyle Ungar demonstrate in Large Language Models Show Human-like Social Desirability Biases in Survey Responses, advanced models like GPT-4 "realize" they are being evaluated and skew their responses to appear more socially attractive—scoring higher in extraversion and lower in neuroticism than a real human would. They aren't simulating a person; they are simulating a "good student" taking a test.
Behavioral vs Surface Alignment
To move beyond surface-level mimicry, researchers have turned to behavioral economics. In Can Large Language Model Agents Simulate Human Trust Behavior?, Chengxing Xie uses "Trust Games" to see if agents mirror human reciprocity and risk perception. While GPT-4 shows high behavioral alignment in these games, other studies reveal a "strategic gap."
Caroline Wang and Kim Stachenfeld found in Discovering Differences in Strategic Behavior between Humans and LLMs that frontier models are often more strategic and exploitative than humans. They use complex internal models to win, whereas humans might rely on intuition or social norms. This makes them "superhuman" in a way that actually makes them worse as representative social proxies.
The Alignment Simulation Tradeoff
Perhaps the most counter-intuitive finding in this field is the alignment-simulation tradeoff. We typically assume that as models get "better" (more instruction-tuned and helpful), they become better at everything. But Joy He-Yueya and Noah Goodman show in Psychometric Alignment: Capturing Human Knowledge Distributions via Language Models that smaller, less "aligned" models often mimic human knowledge patterns better.
Larger models are "unrealistically advanced"—they don't make the same types of mistakes that a human student or voter would. This is formalized in SimBench, which notes that instruction tuning systematically degrades a model's ability to represent the "high-entropy" (messy and diverse) opinions found in real populations.
Grounding the Digital Twin
For LLMs to be useful in social science, we must move from assuming alignment to verifying it. Frameworks like Concordia, introduced by Alexander Vezhnevets and John Agapiou in Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordia, attempt to ground agent behavior in "common sense" and environmental constraints.
Ultimately, the goal is not to build a system that "gives a damn," as Thomas Edelson argued in his early work Can a System Be Intelligent if It Never Gives a Damn?, but to build a system that accurately models the passions and biases of those who do. Until we can distinguish between a model's genuine social reasoning and its desire to be a "helpful assistant," these silicon participants remain a promising, but unvalidated, mirror of our own society.