Large language models are increasingly used as stand-ins for people. Researchers prompt them with a person's description to pre-test surveys, forecast how a population will react to a product, or summarize what "society" thinks. Privacy is where this practice is most likely to mislead. The best-documented fact about human privacy behavior is the privacy paradox: what people say about privacy ("I care a lot about my data") predicts poorly what they actually do (they share it anyway when it is convenient). The models doing the simulating are trained mostly on what people say. We conjecture that LLM simulators reproduce the stated self and miss the behaving self. Concretely, a simulated user's decisions follow its stated privacy attitude much more tightly than a real user's decisions do. We call this attitude over-coupling. If it holds, it is a specific, measurable, and possibly correctable reason why simulated populations fail on privacy. It would also explain why the best of nine frontier models matches only 40.4% of individual decisions in PrivacySIM.
This note proposes to measure over-coupling directly and to test two ways of removing it on held-out studies. The cheap way is a statistical correction fitted on a few real users. The mechanistic way is to re-pose the decision the way it arises in life: as an interruption to a task the simulated person is trying to finish, rather than as a calm question about their values. The comparison that matters is which of these transfers to a privacy study the method has never seen. Both are judged against a plain discrete-choice regression, which a skeptic would reasonably expect to win.
Core Claim
Several findings already point toward over-coupling. PrivacySIM found that stated attitudes alone are a weak predictor of real choices. Its hardest users to simulate were those with high chatbot experience and low stated concern, which is the population a stated-attitude-driven simulator would get wrong. When Groschupp et al. gave LLMs 307 users' written privacy statements and asked them to decide Android permissions, the models enforced the statements more strictly than the users themselves would have. Outside privacy, the GIDEA platform reproduced 10 human–assistant user studies at high semantic similarity. But its simulated people accepted interruptions that real people refused out of fatigue. In each case the simulator acts on the person's declared disposition and ignores the frictions that make real behavior drift away from it.
We make this precise with one model. For user $i$ facing data-sharing scenario $s$, let $y_{is}=1$ if the user shares. Let $a_i$ be the user's stated privacy-attitude score, and let $c_s$ be the scenario's contextual features. Following contextual integrity theory (the view that privacy means information flowing according to the norms of its context), $c_s$ codes data type, recipient, and purpose. Fit
$$P(y_{is}=1) = \sigma(\alpha + \beta\, a_i + \gamma^\top c_s),$$
where $\sigma$ is the logistic function. Fit it once to real decisions, giving $\beta_{\text{human}}$, and once to a simulator's decisions on the same users and scenarios, giving $\beta_{\text{sim}}$. The core claim is that $\beta_{\text{sim}}/\beta_{\text{human}}$ is substantially greater than one across models and studies. The second claim is that this ratio, together with a shift in the overall sharing rate $\alpha$, accounts for a large share of the simulator's individual-level error.
This decomposition separates three failure modes the field currently lumps together under one accuracy number. In the first, the simulator is wrong about how often people share overall, which is a bias in $\alpha$. In the second, it is wrong about how much attitudes matter, which is over-coupling in $\beta$. In the third, it has no individual signal at all, which is the residual. Each calls for a different fix.
Research Program
The first stage is diagnostic. Using the released PrivacySIM suite (1,000 users from five published studies on healthcare consultations, conversational agents, and chatbots), we will code each scenario's contextual parameters and refit the model above per study, per simulator, and per persona condition. We will reuse PrivacySIM's own facet subsets: demographics, prior experience, and stated attitudes. One caveat must be stated up front. PrivacySIM's "actual behavior" comes from user studies, not passive field logs. What we measure is therefore the gap between attitude questionnaires and in-study choices, which is the classic operationalization of the paradox but not deployment behavior.
The second stage builds the synthetic population. We will build two versions. The first is the explicit trait-list personas PrivacySIM uses. The second is narrative backstories in the style of Anthology, which generates open-ended life stories from base models and matches them to real survey respondents; it improved distribution matching by up to 18% over trait lists on Pew surveys. Our backstories will be matched to each PrivacySIM user's recorded attributes. Anthology was validated only on opinion surveys, which are themselves stated responses. Our question is whether richer narratives reduce over-coupling or merely produce more consistent stated selves, which could make it worse.
The third stage is situated elicitation, the main intervention. Instead of "Given your attitudes, would you share X with Y?", the simulated person is placed mid-task. They have a primary goal, such as getting a medical answer or booking a ride, and a time or attention cost. The data-sharing request arrives as an obstacle to that goal, and declining has a stated functional cost. This design follows the privacy literature's account of the paradox. schraefel et al. describe consent interfaces as friction against a primary goal. Wilde et al. argue that disclosure often happens as habitual operations rather than deliberate actions. Work on temporal discounting describes concrete immediate benefits set against diffuse later harms (Krsek et al.).
We will vary two components separately. The first is goal framing, meaning the task and the cost of declining. The second is load, meaning a competing secondary task in the same context. Load is motivated by Kang et al., who found that multitasking dropped trained users' phishing accuracy from 86.4% to 70.4%. We deliberately avoid adding chain-of-thought deliberation as the "realism" knob. Can LLMs Keep a Secret? found step-by-step reasoning sometimes increased leakage in social privacy settings. That is a warning that deliberation changes privacy behavior in ways that are not human-like.
This work differs from its nearest neighbors in its target. PrivacyLens found an "awareness-action gap" in LLM assistants: 98.99% probing accuracy but 25.68% leakage when acting. That is a gap inside the model as an agent, not a model of human users. CI-RL trains assistants to leak less, which is the opposite objective from faithfully simulating people who sometimes share too much. PrivacySIM established the accuracy ceiling but did not decompose it or change the elicitation format. The seed papers supply the stakes. SAGE queries simulated reference populations for perspectives, and Hazra et al. use simulated personas to pre-test behavioral studies. If simulators are over-coupled, both would systematically report what people say about privacy rather than what they do.
Baselines
The naive reference point is PrivacySIM's no-persona condition together with a majority-class predictor per scenario. These show how much any persona information buys.
The strongest existing simulation method is PrivacySIM's best configuration: the strongest frontier model conditioned on all three persona facets. Its reported figure is 40.4% individual-level accuracy (PrivacySIM). We will rerun it with the released prompts on the same models we use, so that differences are not due to model version. Beating it means a statistically reliable gain in held-out accuracy and macro-F1 under the same model and the same persona information.
The strongest non-LLM competitor is a discrete-choice regression. This is the logistic model above, extended with demographic and prior-experience covariates and fitted on real decisions. We found no reported result on Lacuna for regression on PrivacySIM. Nearby evidence makes it a serious threat. Quantifying the Persona Effect found that the best persona-prompted model (Tulu-2-dpo-70b) captured 81% of the variance achievable by linear regression on ANES 2012, without exceeding it. Capturing Human Cognitive Styles found that a logistic model on discourse features reached AUC 0.81 against 0.56 for zero-shot LLMs. For fairness, the regression gets the same persona fields and the same labeled-user budget as any calibrated simulator, and the evaluation uses the same leave-one-study-out splits.
The key ablation is post-hoc coupling calibration. We take the static-persona simulator's sharing log-odds and refit only an intercept and an attitude slope on the labeled budget. This isolates whether situated elicitation adds anything beyond correcting $\alpha$ and $\beta$ after the fact. A second ablation separates goal framing from load, and a third crosses both with trait-list versus backstory personas.
Why not just…?
Why not just fit a regression and skip the simulator? In-distribution, a regression may well win, and we will report it if it does. The case for a simulator rests on held-out studies, whose scenarios differ in domain and wording from anything fitted. There a regression has only coded contextual features to go on, while a simulator can read the scenario itself. Leave-one-study-out evaluation settles this directly. If the regression still matches or beats every simulator on held-out studies, the honest conclusion is that LLM simulation adds nothing for privacy decisions at this granularity, and we will say so.
Why not just recalibrate the static simulator's output? This is the most serious threat, so it is built in as the key ablation. If over-coupling is a stable property of a model, then fitting one slope and one intercept on four studies should transfer to the fifth, and situated elicitation is unnecessary. That would still be a useful finding: a two-parameter correction for privacy simulation. Situated elicitation earns its place only if the calibration constants vary across studies while situated simulators transfer without refitting. Comparing calibration constants across folds tells us which case holds.
Why not just run the human study? Human studies remain ground truth, and this proposal depends on them. But the use cases that motivate simulation, such as pre-testing designs (Hazra et al.) or querying populations at inference time (SAGE), are ones where no human data exist yet. The question is whether simulators can be trusted there, and over-coupling is a concrete reason to think they currently cannot.
Why not just use richer backstory personas? Anthology's gains were measured on stated survey answers, which is exactly the layer we suspect is over-represented. The persona-type ablation tests whether backstories lower $\beta_{\text{sim}}/\beta_{\text{human}}$ or raise it.
Proposed Experiments
The primary experiment is leave-one-study-out evaluation over PrivacySIM's five source studies. For each fold we will run several open and closed models under every combination of persona type, elicitation format, and calibration. We will report individual accuracy, macro-F1, sharing-rate bias, the coupling ratio $\beta_{\text{sim}}/\beta_{\text{human}}$ with bootstrap intervals, and accuracy in the high-experience/low-concern cell that PrivacySIM identified as hardest.
The first decisive check is the diagnosis itself. If the coupling ratio is near one for static simulators, the core claim is false. In that case the 40.4% ceiling reflects missing individual signal, not mis-weighted attitudes, and we redirect toward richer persona inputs.
The second decisive check is that the fix does not break the stated layer. We will simulate the factorial vignette survey of Silber et al., in which 746 German respondents rated willingness to share health data, and check whether simulated stated willingness recovers its reported recipient effects. Those effects are $\beta=-0.660$ for private companies and $\beta=-0.380$ for public health agencies, both relative to universities. A good simulator should match these effects when asked for attitudes and still show human-like decoupling when acting.
The third check is external. We will test whether the load manipulation reproduces the direction of the Kang et al. result, where trained vigilance collapses under multitasking. We will rebuild their email-sorting setup from the published description and compare against their aggregate figures. Only the sign and rough size of the drop are claimed, since individual data are not available to us.
Finally, as a transfer probe, we will apply PrivacySIM-calibrated simulators to the Groschupp permission dataset (14,682 decisions). Its reported generic-LLM agreement with user majorities is 75–89%. We will test whether reducing over-coupling moves individual predictions toward the users' actual, more permissive choices without losing majority agreement.
Workshop authors to connect with
- For Questions of Ought, AI Could Use Some SAGE Advice: Smitha Milli, Ratip Emin Berker, Sonja Kraiczy, Claudia Shi, Jack Kussman, Avinandan Bose, Himaghna Bhattacharjee, Edith Elkind, Ariel D. Procaccia, Maximilian Nickel
- Optimizing Behavioral Experiment Design with In-Silico Simulations: Sanchaita Hazra, Pao Siangliulue, Bodhisattwa Prasad Majumder, Peter Clark, Amy X Zhang, Joseph Chee Chang
- PrivacySIM: Evaluating LLM Simulation of User Privacy Behavior: James Flemings, Murali Annavaram
References
- PrivacySIM: Evaluating LLM Simulation of User Privacy Behavior
- Can LLMs Make (Personalized) Access Control Decisions?
- Design and Evaluation of Generative Agent-based Platform for Human-Assistant Interaction Research: A Tale of 10 User Studies
- Virtual Personas for Language Models via an Anthology of Backstories
- The internet of things: interaction challenges to meaningful consent at scale
- EINS PRIME - perception and realisation of information privacy using measurements and ethnography
- Measuring, Modeling, and Helping People Account for Privacy Risks in Online Self-Disclosures with AI
- Understanding Security Behavior of Real Users: Analysis of a Phishing Study
- Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory
- PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action
- Contextual Integrity in LLMs via Reasoning and Reinforcement Learning
- For Questions of Ought, AI Could Use Some SAGE Advice
- Optimizing Behavioral Experiment Design with In-Silico Simulations
- Quantifying the Persona Effect in LLM Simulations
- Capturing Human Cognitive Styles with Language: Towards an Experimental Evaluation Paradigm
- A preregistered vignette experiment on determinants of health data sharing behavior