Abstract
Traditional social science research relies heavily on questionnaire-based surveys, a method that is increasingly challenged by high costs, slow data collection, and declining response rates. This paper introduces LLM-S³, a comprehensive benchmark designed to evaluate the capability of Large Language Models (LLMs) to serve as virtual survey respondents. By testing models on predicting missing attributes and generating full synthetic datasets, we find that top-tier models like GPT-4 show significant potential, though their success is contingent on precise context-aware prompting and model scale.
Introduction
The "traditional survey pipeline" is facing an operational crisis. High costs and lengthy collection periods hinder the timely delivery of insights necessary for public policy and social research. While the concept of "silicon sampling"—using AI to simulate human responses—has been explored, previous efforts often lacked a systematic, cross-domain evaluation framework. This research addresses this gap by asking: Can an LLM accurately simulate the responses of specific demographic groups, and can it generate statistically representative populations that mirror real-world census distributions?
Methodology: The LLM-S³ Framework
The researchers propose two primary simulation paradigms to evaluate LLM performance:
- Partial Attribute Simulation (PAS): Models are provided with a partial respondent profile (e.g., age, gender, race) and must predict missing behavioral or demographic traits.
- Full Attribute Simulation (FAS): Models are tasked with generating entire synthetic datasets. This is evaluated in two modes: Zero-Context (relying on internal knowledge) and Context-Enhanced (guided by specific population distributions).
The LLM-S³ Benchmark
The study utilizes a massive new benchmark suite, LLM-S³, spanning 11 real-world public datasets across four sociological domains:
- Social & Public Affairs: ANES, GSS, and BIS.
- Work & Income: EmpS and ACS.
- Household & Behavior: RECS, NHTS, and Trell SMU.
- Health & Lifestyle: MxMH, YPS, and MHD.
The evaluation included OpenAI’s GPT-3.5 Turbo and GPT-4 Turbo, alongside Meta’s LLaMA 3.0 and 3.1 (8B).
Key Results and Analysis
The evaluation revealed several critical insights into how different model architectures handle sociodemographic simulation.
Performance Comparison
GPT-4 Turbo emerged as the gold standard, consistently outperforming other models, particularly in complex FAS tasks requiring superior contextual utilization. However, the performance varied significantly based on the task type.
As shown in the figure above, while GPT models generally excelled at multiple-choice tasks, LLaMA variants demonstrated competitive accuracy in predicting continuous numerical values, such as income or travel distance.
Contextual Anchoring and the Overthinking Paradox
A key finding was the "Contextual Anchoring" effect: providing specific contextual cues (e.g., specific years or policy environments) significantly boosted accuracy, transforming LLMs from statistical pattern extractors into contextually grounded reasoning systems.
Conversely, the researchers observed an "Overthinking Paradox." While reasoning-optimized models helped smaller 8B models, they occasionally degraded the performance of larger 70B models. These larger models appeared to engage in excessive internal deliberation, which impaired their ability to simulate straightforward human responses.
Limitations and Future Directions
Despite the promise of LLMs, structured output generation remains a bottleneck. Models—particularly LLaMA variants—often struggled to maintain the requested JSON formats or row counts. Furthermore, while context-enhanced generation improves results, it currently requires prior distribution data. Future research will focus on simulating "unobserved" or "hard-to-reach" populations and refining alignment strategies to mitigate training data biases in synthetic populations.