PrivacySIM: Evaluating LLM Simulation of User Privacy Behavior

James Flemings, Murali Annavaram

TL;DR

Can Large Language Models (LLMs) act as "silicon doubles" for real people when it comes to sensitive privacy decisions? This paper introduces PrivacySIM, a benchmark that tests LLMs against the ground-truth responses of 1,000 real users. The results are a reality check: even the best models top out at ~40% accuracy, struggling to navigate the "privacy paradox" where what users say and what they actually do often diverge.

What Problem Does This Paper Address?

Researchers and developers are increasingly using LLMs to simulate human behavior—a practice sometimes called "silicon sampling." This is particularly attractive for privacy research, where conducting large-scale human studies is slow, expensive, and can lead to "privacy fatigue" for participants.

However, privacy behavior is notoriously difficult to model. Humans are inconsistent; we might claim to be deeply concerned about data usage while simultaneously sharing sensitive health info with a chatbot for the sake of convenience. This paper asks: Can we drive an LLM to simulate a specific individual's privacy choices using a core set of persona attributes (like age, tech experience, and stated attitudes)?

Methods

The authors developed PrivacySIM, an evaluation suite that bridges the gap between LLM simulation and real-world data.

Figure 1: Overview of PRIVACYSIM, an evaluation suite for simulating user privacy behavior. We collect user responses and questionnaires from existing user studies on privacy behavior in LLM and AI contexts. We then condition LLMs on subsets of a user’s privacy persona (demographics, previous experiences with LLMs, and stated privacy attitudes) to simulate their responses to data-sharing questions. Finally, we evaluate the simulation quality by comparing the LLM’s simulated responses to the user’s actual responses.

The core methodology involves:

  1. Persona Conditioning: Prompting LLMs with three "facets" of a user's identity:
  2. Demographics ($D_u$): Age, gender, education.
  3. Previous Experiences ($E_u$): Familiarity with AI and chatbots.
  4. Stated Privacy Attitudes ($A_u$): General concerns about data usage.
  5. Simulation: Asking nine frontier LLMs (including GPT-5.4, Gemini 3.1 Pro, and Claude 4.6) to answer data-sharing questionnaires as if they were that specific user.
  6. Scoring: Using "tolerance accuracy" to compare LLM responses to ground-truth human data. Since some studies used 0-100 scales, the metric credits a "match" if the LLM is within $\pm 15$ points of the human response.

Datasets

The authors curated a dataset from five published user studies on privacy behavior, spanning domains like healthcare consultations, AI agent permissions, and group chatbots.

  • Scale: 1,000 unique users and 9,400 user-question pairs.
  • Diversity: The studies cover different countries and use various response formats (Likert scales vs. numeric scales).
Table 1: Summary of source user studies in PRIVACYSIM. Du, Eu, and Au denote demographics, previous experiences, and privacy attitudes, respectively. “Domain” refers to the data-sharing context posed in the original study.

Key Insights and Ideas

  • The Privacy Paradox is the Main Boss: Stated attitudes ($A_u$) alone are poor predictors of behavior. In fact, conditioning on attitudes alone often decreased simulation accuracy compared to using previous experience data. LLMs struggle when a user's stated "concern" doesn't match their "sharing" behavior.
  • LLMs Have a "Generic" Bias: Without any persona info, LLMs default to a "moderate" privacy stance based on general population norms. While this provides a non-trivial baseline, it systematically fails to capture the more restrictive behavior of highly privacy-conscious users.
  • Experience Trumps Attitude: A user's familiarity with AI ($E_u$) is a more reliable signal for the model than what the user says they care about.
  • The "Hardest" Users: The most difficult group to simulate are "low-stance, high-exposure" users—people who use AI a lot but claim not to care about privacy. The LLM often overestimates how much they are willing to share.

Key Results

  • Performance Ceiling: The top-performing model, Gemini 3.1 Pro, achieved only 40.4% accuracy when conditioned on experience and attitudes.
  • Modest Gains from Personas: Adding persona facets consistently improves simulation quality over a "no-persona" baseline, but the boost is relatively small (roughly 3-4 percentage points).
  • Reasoning Doesn't Help (Yet): Interestingly, increasing "reasoning effort" (test-time compute) or using larger model variants within the same family yielded only negligible improvements in accuracy.
Figure 2: Average accuracy by prompt type and model across user studies. Du, Eu, and Au denote demographics, previous experiences, and stated privacy attitudes (Section 3.2). Solid bars are the average of all five user studies; hatched bars are the average of the three user studies that contain demographics. The black diamond line is the average across the eight models on the x-axis. For the prompt types containing demographics, the bars appear only as hatched. Our key findings show that (1) Gemini 3.1 Pro achieves the highest accuracy (40.4%) on Eu+Au across the five studies, a 4-percentage-point gain over the no-persona baseline. (2) The no-persona prompt (none) achieves a non-trivial simulation accuracy, suggesting that LLMs can simulate user behavior to some extent even without conditioning on persona facets. (3) The simulation quality can degrade when conditioning only on privacy attitudes compared to previous experiences, consistent with the privacy paradox.
Table 3: Simulation accuracy across model families and variants. For GPT-5.4 we report the smaller mini model and the full model un- der low and medium reasoning effort. We find that reasoning effort has negligible impact and model size can yield modest simulation im- provement.

Limitations and Future Work

The authors acknowledge that their current persona facets (demographics, experience, attitudes) are likely insufficient for high-fidelity simulation. The "gap between what users say and what they actually do" remains the central obstacle.

Future directions include: * Eliciting richer behavioral signals beyond simple survey responses. * Developing new user studies specifically designed to capture the nuances needed for LLM simulation. * Exploring whether LLMs can be used for low-stakes, early-stage design tasks (like surfacing obvious privacy violations) even if they aren't yet ready to replace human subjects in high-stakes validation.

Summary updated

Abstract

Large language models (LLMs) are increasingly used to simulate human behavior, but their ability to simulate $individual$ privacy decisions is not well understood. In this paper, we address the problem of evaluating whether a core set of user persona attributes can drive LLMs to simulate individual-level privacy behavior. We introduce PrivacySIM, an evaluation suite that benchmarks LLM simulation of user privacy behavior against the ground-truth responses of 1,000 users. These users are drawn from five published user studies on privacy spanning LLM healthcare consultations, conversational agents, and chatbots. Drawing on these user studies, we hypothesize three persona facets as plausible predictors of privacy decision-making: demographics, previous experiences, and stated privacy attitudes. We condition nine frontier LLMs on subsets of these three facets and measure how often each model's response to a data-sharing scenario matches the user's actual response. Our findings show that (1) privacy persona conditioning consistently improves simulation quality over no-persona conditioning, but even the strongest model (40.4\% accuracy) remains far from faithfully simulating individual privacy decisions. (2) A user's stated privacy attitudes alone may not be the best predictor because they often diverge from the user's actual privacy behavior. (3) Users with high AI/chatbot experience but low stated privacy attitudes are the most challenging to simulate. PrivacySIM is a first step toward understanding and improving the capabilities of LLMs to simulate user privacy decisions. We release PrivacySIM to enable further evaluation of LLM privacy simulation.
BibTeX
@inproceedings{flemings2026privacysim,
  author = {James Flemings and Murali Annavaram},
  title = {{PrivacySIM: Evaluating LLM Simulation of User Privacy Behavior}},
  booktitle = {Social Sim @ COLM},
  year = {2026},
  url = {https://openreview.net/forum?id=drBP7NzytR},
}
Figure