Necessary for What? Auditing Demographic Disclosure to LLM Synthetic Respondents

Aspen Wang, Yen Yu Shan, Lun-Wei KuarXiv 2026 · arXiv

TL;DR

Researchers audited whether feeding real demographic data into LLMs actually makes them better "synthetic survey respondents." They found that while personal records are often unnecessary for population-level estimates, they are required for group comparisons—yet even then, LLMs tend to rely on stereotypes ("label-following") and flatten complex human attitudes, especially regarding sensitive topics like religion and ethnicity.

What Problem Does This Paper Address?

In the burgeoning field of "synthetic silicon sampling," researchers use Large Language Models (LLMs) to act as "digital twins" for survey panelists. By prompting a model with a person’s demographic profile (age, race, education) and their previous answers, researchers hope to predict how that person would answer new questions.

However, this practice creates a significant privacy risk. Combining just a few demographic attributes can often uniquely identify an individual in a dataset. The principle of data minimization dictates that we should only use personal data if it is strictly necessary for the task. This paper asks: Is disclosing a respondent's personal record to an LLM actually necessary for accurate survey results, or are we risking privacy for no gain?

Methods

The authors conducted a systematic audit by varying the demographic information provided to 11 open-weight models (like Llama 3.1, Qwen, and Mistral) and 3 hosted models (GPT-4, Claude). They tested several conditions:

  • Own Record: The model gets the respondent's actual 15-field demographic profile.
  • Shuffled: The model gets a real profile, but from a different random respondent.
  • External/Synthetic: The model gets a generic profile from census data (PUMS) or a synthetic persona (Nemotron).
  • Reduced Granularity: The number of fields was stripped down from 15 to as few as 4.

Crucially, they compared these LLM outputs against a statistical reference model (logistic regression) to see if the LLMs were actually "reasoning" or just performing basic statistical matching. They also introduced a "regrouping test" to detect label-following—checking if the model's answers changed because of the person it was mimicking or simply because of the demographic label in the prompt.

Datasets

The study utilized two major U.S. surveys:

  1. Pew American Trends Panel (Wave 54): Focused on economic inequality (1,000 respondents, 63 questions).
  2. International Social Survey Programme (ISSP 2023): Focused on national identity and citizenship (1,513 respondents, 27 questions).
  3. External Benchmarks: 2019 American Community Survey (ACS) PUMS data and the Nemotron-Personas-USA dataset were used to provide non-personal "substitute" profiles.

Key Insights and Ideas

  • The "Label-Following" Trap: LLMs often ignore the individual's specific history and instead "hallucinate" answers based on the demographic labels provided. If you tell a model it is a "high-income" individual, it will answer like a stereotypical wealthy person, even if the specific individual's previous answers suggested otherwise.
  • Circumstances vs. Attitudes: Models are surprisingly good at maintaining differences in circumstances (e.g., wealthy people are more likely to have stocks). However, they are terrible at capturing attitudes (e.g., how religious people feel about government responsibility). They tend to "flatten" these views, making different groups look much more similar than they actually are.
  • The Privacy Paradox: The most sensitive categories of data—religion and ethnic origin—resulted in the least faithful synthetic responses. We are disclosing the most private information to get the most inaccurate results.
  • Population vs. Group Estimates: If your goal is just to see how the general population feels, you don't need real personal records at all; generic census personas work just as well. You only "need" the real record if you are comparing specific subgroups (e.g., Young vs. Old), but even then, the results are often biased by stereotypes.

Key Results

  • Minimal Gain for Individuals: Adding a demographic record to a prompt that already contains eight previous answers only improved individual prediction accuracy (Brier score) by a measly 1% to 3%.
  • Significant Flattening: The differences LLMs produced between groups were only about one-fifth (0.21 ratio) as large as the differences found in real human data.
  • Resolution Errors: When the models did try to resolve group differences, they pointed the "wrong way" (reversed the actual human trend) about one in nine times. For religious groups, this error rate jumped to 25%.
  • Identifiability: The authors found that while reducing a profile to 4 fields helps, the combination of those fields plus previous survey answers still made 90% of respondents unique (and thus potentially re-identifiable).

Limitations and Future Work

  • U.S. Centricity: The study focused exclusively on U.S. surveys; LLM behavior might differ significantly in other cultural or linguistic contexts.
  • Prompt Sensitivity: The research used a specific prompt format. Future work could explore if "system messages" or different elicitation techniques (like Chain-of-Thought) reduce label-following.
  • The "Audit" Path: The authors suggest that survey organizations should run "audits" on questions they have already fielded to decide which future estimates actually require demographic disclosure.
  • Model Scaling: Interestingly, larger models did not solve the "flattening" problem; they often just became more confident in their stereotypical "label-following."

For those interested in the code or running their own audit, the authors have released their framework at github.com/aspenywang/necessary-for-what.

Summary updated

Abstract

Large language models (LLMs) increasingly answer surveys on behalf of real panelists, and each panelist's recorded demographic attributes (e.g., age, education, ethnicity) are often placed in the prompt on the premise that richer profiles make more faithful synthetic respondents. While data minimization limits personal data to what a purpose requires, a synthetic survey serves several purposes at once, from predicting individual answers to estimating how opinions are distributed and comparing groups. We ask which of these purposes require the respondent's record, and whether improved estimates support sound conclusions about groups. We vary whose profile a model receives, where it comes from, and how many fields it contains, in two U.S. surveys with eleven open-weight and three hosted models, and compare each model with a statistical model fit to the same information. We make four contributions. First, we show that the record adds little to individual predictions and that, for the open-weight models, most of its gain in population estimates remains when it is replaced by census data or a synthetic persona. Second, we find that group comparisons improve only with the respondent's own record, and that even then the models' group differences rest more on the demographic label in the prompt than on the individual respondent, which the usual check of reproducing known group differences does not reveal. Third, we find that the models make groups look more alike than they are, keeping differences in circumstances while flattening or reversing differences in attitudes, most sharply for religious affiliation and ethnic origin, both special categories of personal data. Fourth, we outline an audit that a survey organization can run on questions it has already fielded to decide which estimates may use the record.
BibTeX
@misc{wang2026necessary,
  author = {Aspen Wang and Yen Yu Shan and Lun-Wei Ku},
  title = {{Necessary for What? Auditing Demographic Disclosure to LLM Synthetic Respondents}},
  year = {2026},
  eprint = {2611.20606},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20606},
}
Figure