Researchers increasingly ask large language models (LLMs) to stand in for human survey respondents: "answer as a 45-year-old religious woman in Ghana would." Benchmarks report that the best models reproduce a meaningful share of real human answer distributions. These are aggregate answer distributions, meaning the share of a group that picks each option. Many come from public surveys such as the European Social Survey and Afrobarometer, whose results are also published on the web in reports and summary tables. This proposal tests a simple suspicion. Part of what looks like simulation may be recall of published numbers. If so, that recall is unevenly distributed across groups. Broad, well-reported groups would look faithfully simulated. Small or marginalized intersections that no one ever tabulated would show the stereotyped fallback that bias research keeps finding.
The core move is to split evaluation cells by provenance, not at random. Each pairing of a survey question with a demographic group is a cell, and each cell gets a label. A cell is "published" if its distribution appears in official reports or summary tables. It is "computable but untabulated" if it can only be computed from respondent-level microdata. It is "post-cutoff" if it comes from a survey wave released after the model's training cutoff. We then ask three questions. Does fidelity drop on cells the model could not have read? Does a model's ability to recite a published figure predict how well it "simulates" that cell? On unread cells, do errors lean in the direction of known stereotypes? If the answers are yes, current fidelity scores overstate simulation ability and hide bias amplification. Both conclusions bear directly on whether silicon subjects can be used in social science.
Why this is the right question now
The motivation comes from a tension inside the simulation literature. SimBench merges 20 behavioral datasets, including Afrobarometer, Latinobarómetro, the European Social Survey and OpinionQA, into one scored benchmark. Its best model, Claude 3.7 Sonnet, reaches 40.80 out of 100. Its fidelity score correlates with the MMLU-Pro knowledge benchmark at r = 0.939. SimBench reads this as a sign that general capability drives simulation. An equally consistent reading is that simulation fidelity is partly knowledge of published polling. SimBench never separates the two readings, because it does not control for whether the target distributions were themselves on the web.
The contamination literature gives reasons to take the second reading seriously, and it also limits how to test it. DyVal generates fresh reasoning problems from graphs. Models with strong static scores collapse to near zero on these problems, which shows that static benchmarks can hide memorization. Vision-Language Bootstrapping finds up to 84.46% overlap between static multimodal benchmarks and pretraining corpora. MM-DETECT shows that leakage into multimodal models often enters through the text pretraining of the base LLM.
All three methods perturb or regenerate the test item. That works for arithmetic and visual question answering, where the answer follows from the item. It does not work for population simulation. Here the "answer" is an empirical fact about a group, so a reworded question still points to the same memorized percentage. The contamination unit is the aggregate statistic, not the item text. Provenance stratification is designed for that unit. This is the main way the proposal differs from all three works.
Bias amplification supplies the second half of the claim. Quantifying Risk of Epistemic Harm from the Use of AI Surrogates finds a "privilege bump." LLM surrogates rate dominant groups more warmly than human raters do and penalize marginalized ones. Their divergences from human responses are 1.5 to 2 times the variation among humans, and all five tested models fail a 4/5ths fidelity-parity rule. SimBench likewise finds the worst fidelity for religious and ideological groups. Uncovering Stereotypes in LLMs shows models falling back on stereotypes as tasks get harder. FairBelief shows that harmful associations sit at the most likely outputs, so changing decoding settings does not remove them.
We propose a mechanism that links these findings to contamination. When a group's numbers were published, the model can recall them. When they were not, it has to fill in, and it fills in with stereotypes. Under this account, part of the reported fidelity gap between groups reflects which groups get tabulated, not only which groups the model understands. The epistemic-harm study tested static text prompts and did not ask this question. We test it directly.
Core Claim
We predict three things. First, fidelity measured with the SimBench score will be reliably higher on published cells than on matched untabulated or post-cutoff cells. Second, a model's accuracy when asked outright to recall a published percentage will predict its simulation fidelity on that cell. Third, on unread cells, the model's signed error relative to the microdata truth will correlate with the group's social status, in the direction of the privilege bump.
If all three hold, much of current simulation fidelity is retrieval, and the remaining ability is biased exactly where social scientists most want surrogates: under-surveyed groups. If none hold, that is also a useful result. It would be the first evidence that silicon-subject fidelity survives a real contamination control.
Research Program
Building provenance-labelled cells. We rebuild ground truth from respondent-level microdata for the public surveys inside SimBench that release such data, starting with the European Social Survey, Afrobarometer and Latinobarómetro. For each question we enumerate group definitions of matching complexity, such as country × religiosity × age band.
Each cell is labelled published if its distribution appears in an official report table or in the survey's public online tabulation tool. Otherwise it is labelled untabulated. Where possible, we add cells from waves released after each model's stated cutoff. Labelling is manual and auditable. Two annotators search the official outputs, and disagreements are resolved conservatively by calling the cell published.
The main confound is that untabulated cells tend to be smaller and more specific. We address this in three ways. We match published and untabulated cells on how many attributes define the group, on respondent count, and on the entropy of the true distribution. We report every result against a bootstrap noise ceiling, defined as the fidelity a second human sample of the same size would achieve. Most importantly, we exploit cases where the same crosstab is published for some countries but not others. Those within-crosstab contrasts are the cleanest comparison we have.
Probing recall directly. For each published cell we ask the model a factual question: "In [survey, round], what share of [group] in [country] answered [option]?" Recall error is then a per-cell memorization score. We relate it to simulation fidelity on the same cell, controlling for the matching covariates.
We also apply the "Unawareness" principle from PIMMUR. That audit found that 50.8% of LLM simulations let the model work out the underlying experiment. We therefore run every simulation prompt in two versions. The named version identifies the survey and wave. The anonymized version presents the item alone. Retrieval predicts that naming the source helps much more on published cells than on untabulated ones.
Measuring stereotype fallback. On untabulated and post-cutoff cells, we compute the signed error between the model's expected answer and the microdata truth on items with a clear valence, such as trust, life satisfaction and attitudes toward institutions. We regress this signed error on a group-status index taken from the Wheel of Power and Privilege attributes used by the epistemic-harm study. We also compute that study's Fidelity Parity Ratio separately on published and unread cells. The prediction is that parity looks better on published cells and fails more severely on unread ones.
A modality-shift arm, kept secondary. Since MM-DETECT finds that leakage flows through text pretraining, one cheap probe of string-level recall is to show vision-language models the survey item as a rendered image instead of text. This is confounded by the multimodal reasoning gap. Plot Twist shows that models misread elementary chart details. Reasoning After Perception shows near-zero performance on perception-heavy tests. So a drop in fidelity could mean the model simply failed to read the question.
We borrow the unit-test logic of Plot Twist to handle this. Before answering, the model must transcribe the rendered item, and we score fidelity only on items transcribed exactly. With perception verified, any remaining drop that appears on published but not on untabulated cells points to retrieval keyed on surface text. This arm is a contamination diagnostic. We make no claim that it measures multimodal social reasoning.
Baselines
The naive reference point is a uniform distribution over options. It scores 0 on the SimBench score by construction, and SimBench reports that 10 of its 45 models fall below it. A second naive reference is the country-level marginal distribution of the question, ignoring group attributes. A model that cannot beat this on untabulated intersections has added no group-specific information.
The strongest baseline that uses no LLM is a statistical estimator. It applies iterative proportional fitting, which adjusts a joint table until it matches known one-way totals, to only the published one-way marginals for each country. This is exactly the information a memorizing model could have absorbed, and it is the most dangerous competitor. If an LLM matches this estimator on untabulated intersections, its "simulation" is indistinguishable from interpolating published tables. The estimator receives the same published sources the annotators identified, and we tune it on a held-out set of published intersections. Beating it means a statistically reliable SimBench-score advantage on matched untabulated cells.
The strongest reported LLM result is Claude 3.7 Sonnet's 40.80 SimBench score from SimBench. We re-run frontier and open models, including base and instruction-tuned pairs, with SimBench's elicitation methods: token probabilities for base models and verbalized percentage distributions for tuned models. All models share prompts and get one tuning pass on a development split.
The key ablation replaces the provenance split with a random split over the same cells. If a random split shows no gap and the provenance split does, provenance is the operative factor. We also compare against an item-perturbation control in the spirit of DyVal and VLB, which paraphrases items and adds irrelevant context. This tests whether existing contamination tools already capture the effect.
Why not just…?
Why not just run MM-DETECT or option-order sensitivity tests? Those tests detect whether a model has seen a particular item and its answer options. Our hypothesis concerns memorized aggregate statistics, which can be learned from a news article that never reproduces the questionnaire. We will run option-order sensitivity on every item and report how it relates to per-cell recall error. A weak relationship would show that item-level detectors miss this channel. A strong one would mean the cheaper tool suffices, and we would say so.
Why not just paraphrase or perturb items, as DyVal and VLB do? A reworded question about trust in parliament still points to the same remembered percentage. Retrieval therefore predicts that paraphrasing barely moves fidelity while the provenance split moves it a lot. The perturbation control measures exactly this. If paraphrasing alone reproduces the provenance gap, the simpler method wins, and our contribution narrows to the bias analysis.
Why not just add chain-of-thought or use a reasoning model? SimBench reports that extra inference compute, including o1-mini, did not improve fidelity. Deliberate reasoning moves models away from intuitive human answers. We include one reasoning-model condition only to check whether reasoning increases retrieval on named prompts. We do not expect it to close the gap on unread cells.
Why not just weight published tables and skip the LLM? This alternative is a real threat, which is why it is the main baseline and not a strawman. If it matches LLMs on untabulated cells, the honest conclusion is that surveys of unmeasured groups gain nothing from silicon subjects beyond what published numbers already give. That conclusion is worth publishing.
Decisive Checks
The proposal succeeds if four results hold. On within-crosstab matched pairs, frontier models show a positive published-minus-untabulated fidelity gap that exceeds the bootstrap noise ceiling. Per-cell recall error predicts fidelity after the matching controls. Naming the survey helps disproportionately on published cells. On unread cells, signed error tracks group status, and the Fidelity Parity Ratio worsens relative to published cells.
The proposal fails cleanly in two cases. One is no gap under the cleanest country-contrast design. The other is a model that beats the published-marginal estimator on untabulated intersections with errors unrelated to status. Either outcome would be the first contamination-controlled evidence that LLM population simulation is more than recall.
References
- SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping
- Both Text and Images Leaked! A Systematic Analysis of Data Contamination in Multimodal LLMs
- Quantifying Risk of Epistemic Harm from the Use of AI Surrogates in Social Science Research
- Uncovering Stereotypes in Large Language Models: A Task Complexity-based Approach
- FairBelief - Assessing Harmful Beliefs in Language Models
- The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- Position: Reasoning After Perception Means Reasoning Without Vision