Does a Simulated Population Reproduce the Privacy Paradox?

People say they care about privacy and then share their data anyway. Researchers call this the privacy paradox. Language models are now used to stand in for human survey respondents, and recent benchmarks score them on whether they predict each person's sharing decision. On that test even the best frontier model reaches only 40.4% accuracy (PrivacySIM). This note argues that per-person accuracy is the wrong test for social simulation. The question that matters is whether a synthetic population reproduces the structure of human privacy behavior. That structure has three parts: which scenarios are hard to say yes to, which scenarios separate cautious people from permissive ones, and how loosely stated attitudes are tied to actual choices. A simulator that matches individuals poorly but reproduces this structure can still support population-level research. A simulator that erases the attitude–behavior gap cannot be used to study the paradox at all, however its accuracy comes out.

The proposal uses item response theory, a family of statistical models that explain each yes/no answer as the interaction between a person's hidden trait and a question's properties. We fit the same two-trait model, one trait for stated privacy attitude and one for disclosure behavior, to real human data and to language-model respondents given the same personas. We then compare the fitted parameters directly. Our central prediction is that language models collapse the paradox: their simulated behavior tracks their simulated attitudes far more tightly than human behavior does. We also predict that much of this coupling comes from a known measurement flaw. When attitude statements sit in the same context window as the decision, they prime it.

Core Claim

The privacy paradox can be stated as one number: the correlation $\rho$ between two hidden traits. For a respondent $u$, let $\eta_u$ be their latent stated privacy concern and $\theta_u$ their latent willingness to disclose. Attitude items $k$ and disclosure scenarios $i$ follow a two-parameter logistic model:

$$P(\text{endorse}_{uk}) = \sigma\big(a_k(\eta_u - b_k)\big), \qquad P(\text{share}_{ui}) = \sigma\big(\alpha_i(\theta_u - \beta_i)\big), \qquad (\eta_u, \theta_u) \sim \mathcal{N}(0, \Sigma_\rho).$$

Here $\sigma$ is the logistic function. $b_k$ and $\beta_i$ are item difficulties, which set how much concern or permissiveness it takes to say yes. $a_k$ and $\alpha_i$ are discriminations, which set how sharply an item separates people on either side of that threshold. $\Sigma_\rho$ is a correlation matrix with off-diagonal $-\rho$, signed so that a positive $\rho$ means more stated concern goes with less sharing. Humans show a weak $\rho$. That weakness is the paradox, measured after the model has removed the noise that comes from asking only a few questions.

We claim three things. First, language-model respondents yield $\hat\rho$ well above the human value. Second, $\hat\rho$ drops substantially when attitudes and decisions are elicited in separate sessions. Third, matching human item difficulties and discriminations is a better criterion for simulator quality than individual accuracy. Under the third claim, simulators can rank differently than they do on accuracy.

The work builds on several findings and departs from them in specific ways. PrivacySIM supplies the human ground truth and the motivating observation that stated attitudes predict actual decisions poorly. It scores only exact matches and never asks whether the simulator reproduces the weak link itself. Can LLMs Make (Personalized) Access Control Decisions? provides the strongest hint for our prediction. Its models enforced users' written privacy rules more strictly than the users themselves did, which is exactly the over-coupling we expect, but it was framed as a desirable property of a privacy assistant rather than a fidelity failure of a simulator. On the validity side, Rethinking Psychometric Evaluation of LLMs showed that self-report-to-behavior correlation for sycophancy falls from +0.47 within a session to −0.07 across sessions. Carryover Effect in LLM Personality Measurement showed that personality text in prior context shifts later answers even when the user disowns it. Both findings supply the mechanism we test. Both were established on the model's own traits, not on a model simulating a human whose attitudes and actions should diverge. The Personality Illusion found that persona prompts move self-reports strongly but barely move behavior. This seems to contradict our prediction of over-coupling. We reconcile the two below, and the reconciliation is itself testable.

Research Program

The primary data source is the Android permission dataset from Can LLMs Make (Personalized) Access Control Decisions?. It contains 307 free-text privacy statements and 14,682 allow/deny decisions over 111 permission tasks in 14 apps. That is a dense person-by-item matrix of the kind item response theory needs.

We code each statement into a small set of binary attitude indicators, such as "objects to location sharing" or "accepts sharing for core functionality." Two human annotators do this coding, and we report their agreement, giving the attitude side of the model. The 111 tasks form the behavior side.

The second source is the five studies pooled in PrivacySIM, which hold 1,000 users with stated attitudes and data-sharing responses. We do not know from the available description how many decisions each user contributes. If it is few, those studies support item-level comparisons pooled across users rather than per-person traits, and we will report them that way.

For simulation, each human persona is rendered for each language model in the format PrivacySIM uses, with demographics, prior experiences, and stated attitudes. The model then answers every attitude item and every permission task. We draw several samples per persona so that within-model randomness is not mistaken for population spread. The model pool covers several open-weight families at two or more sizes plus the frontier models available to us. That gives enough distinct simulators to compare how each one ranks on accuracy against how it ranks on structural fidelity.

We fit the model above by Bayesian estimation, separately to the human matrix and to each simulator's matrix, and compare three quantities. The first is $\hat\rho$ with its posterior interval. The second is the Spearman correlation between human and simulated difficulties $\beta_i$. The third is the same correlation for discriminations $\alpha_i$, the parameter LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels found hardest to recover.

The mechanism study crosses two factors. The first is whether the stated-attitude text is present in the decision prompt. The second is whether attitude items and decisions are answered in the same conversation or in independent calls. This factorial design is borrowed from Rethinking Psychometric Evaluation of LLMs.

A third factor follows Carryover Effect. We insert a third-person narrative about a very cautious or very permissive other person before the decisions and check whether the simulated population's mean disclosure trait shifts toward it.

These designs also let us reconcile our prediction with The Personality Illusion. That paper changed a persona's trait label and measured downstream behavior on unrelated tasks. Here the attitude text explicitly names the very behavior being decided, so lexical matching alone can couple the two. If our coupling disappears when attitudes are paraphrased so that they no longer share wording with the permission tasks, both results are explained by the same surface mechanism.

Baselines

The naive reference point is a no-persona simulator. Each model answers every task as a generic user, and the population is the set of sampled answers. This tells us how much structure comes from the scenarios alone. A simulator that matches human difficulties no better than this has learned nothing about people. We run it on identical items with the same number of samples.

The strongest existing approach to the simulation question is full persona conditioning as in PrivacySIM. Its best reported result is 40.4% individual-decision accuracy. We replicate this conditioning on the same users and the same models. The comparison is not a contest to beat 40.4%; it asks whether accuracy and structural fidelity rank simulators the same way. On the Android data, the comparable reference is the personalized model of Can LLMs Make (Personalized) Access Control Decisions?. That paper reports 75% to 89% agreement with user majorities and a 2% to 5% average gain from personalization. We reproduce that agreement metric so readers can place our simulators on its scale.

The strongest existing method for recovering item properties from synthetic respondents is the response-based classical calibration of LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels. It computes item-total correlations over an all-persona pool and reached a Spearman correlation of 0.241 with human discrimination on reading items. No reported result found on Lacuna for this method on privacy items. We compute it on exactly the same simulated responses we feed to the two-trait model. Any gap between the two methods therefore reflects estimation, not data. Beating it means a higher human–simulated discrimination correlation with non-overlapping bootstrap intervals.

The key ablation removes the stated-attitude facet from the persona, keeping only demographics and experiences, and compares $\hat\rho$ and fidelity with and without it. If over-coupling is driven by attitude text, this ablation should lower $\hat\rho$ toward the human value. It may also improve fidelity on the subgroup PrivacySIM found hardest: users with high AI experience and low stated concern.

Why not just…?

Why not just correlate each person's attitude score with their raw sharing rate, without any latent-trait model? This is the most serious alternative, and we run it alongside the model. Raw correlations are weakened by measurement noise in short scales: a handful of coded attitude indicators and an uneven set of permission tasks. The weakening differs between humans and simulators, because simulators answer far more consistently, as The Personality Illusion reports for alignment-tuned models. A raw comparison would therefore make simulators look more coupled partly for reasons unrelated to the paradox. The latent model removes that difference. If the raw and latent comparisons reach the same verdict, we will say so, and the cheaper metric becomes the recommendation.

Why not just match the marginal sharing rate per scenario between humans and simulators? Matching marginals checks difficulty only. A simulator can give every scenario the right yes-rate while assigning those yeses to the wrong people. That is precisely the failure that would erase the paradox. Our comparison of discrimination and of $\rho$ catches this. We report marginal matching as a secondary metric to show cases where it passes and structure fails.

Why not just ask a model to rate how sensitive or difficult each scenario is? Direct estimation of item properties has been tested at scale and does poorly. The best of 42 models reached a Spearman correlation of 0.152 for discrimination in LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels. PrivacyLens shows a related failure in privacy itself: near-perfect answers to privacy-norm questions alongside 25.68% leakage in action. We include direct rating as a cheap comparison arm on the 111 tasks but do not expect it to compete.

Why not just fine-tune the simulator on human decisions? Fine-tuning would likely raise accuracy. But it could also teach the model to reproduce the human $\rho$ for the wrong reasons, or to overfit the 14 apps. Our metrics are what would reveal either problem, so fine-tuning is a downstream use of this work rather than a substitute for it. We leave it out of scope except for one small held-out-app check, if the main findings hold.

Decisive Checks

The central prediction fails if simulated $\hat\rho$ falls inside the human posterior interval for most simulators under standard same-prompt conditioning. That outcome would mean language models already reproduce the paradox, and the accuracy ceiling has some other cause.

The mechanism claim fails if moving attitudes to a separate session, or paraphrasing them away from task wording, leaves $\hat\rho$ unchanged.

The claim that structural fidelity is a distinct criterion fails if simulators rank identically on individual accuracy and on human–simulated parameter agreement across the model pool. We will report Kendall's tau between the two rankings with a bootstrap interval.

Finally, the human fit carries a finding of its own whatever the simulators do. If the human $\rho$ turns out to be large once measurement error is removed, the privacy paradox in these data is partly a product of short attitude scales. Simulation studies that treat it as a fixed target would then need revisiting.

Workshop authors to connect with