TL;DR
Can Large Language Models (LLMs) act as "silicon doubles" for real people when it comes to sensitive privacy decisions? This paper introduces PrivacySIM, a benchmark that tests LLMs against the ground-truth responses of 1,000 real users. The results are a reality check: even the best models top out at ~40% accuracy, struggling to navigate the "privacy paradox" where what users say and what they actually do often diverge.
What Problem Does This Paper Address?
Researchers and developers are increasingly using LLMs to simulate human behavior—a practice sometimes called "silicon sampling." This is particularly attractive for privacy research, where conducting large-scale human studies is slow, expensive, and can lead to "privacy fatigue" for participants.
However, privacy behavior is notoriously difficult to model. Humans are inconsistent; we might claim to be deeply concerned about data usage while simultaneously sharing sensitive health info with a chatbot for the sake of convenience. This paper asks: Can we drive an LLM to simulate a specific individual's privacy choices using a core set of persona attributes (like age, tech experience, and stated attitudes)?
Methods
The authors developed PrivacySIM, an evaluation suite that bridges the gap between LLM simulation and real-world data.
The core methodology involves:
- Persona Conditioning: Prompting LLMs with three "facets" of a user's identity:
- Demographics ($D_u$): Age, gender, education.
- Previous Experiences ($E_u$): Familiarity with AI and chatbots.
- Stated Privacy Attitudes ($A_u$): General concerns about data usage.
- Simulation: Asking nine frontier LLMs (including GPT-5.4, Gemini 3.1 Pro, and Claude 4.6) to answer data-sharing questionnaires as if they were that specific user.
- Scoring: Using "tolerance accuracy" to compare LLM responses to ground-truth human data. Since some studies used 0-100 scales, the metric credits a "match" if the LLM is within $\pm 15$ points of the human response.
Datasets
The authors curated a dataset from five published user studies on privacy behavior, spanning domains like healthcare consultations, AI agent permissions, and group chatbots.
- Scale: 1,000 unique users and 9,400 user-question pairs.
- Diversity: The studies cover different countries and use various response formats (Likert scales vs. numeric scales).
Key Insights and Ideas
- The Privacy Paradox is the Main Boss: Stated attitudes ($A_u$) alone are poor predictors of behavior. In fact, conditioning on attitudes alone often decreased simulation accuracy compared to using previous experience data. LLMs struggle when a user's stated "concern" doesn't match their "sharing" behavior.
- LLMs Have a "Generic" Bias: Without any persona info, LLMs default to a "moderate" privacy stance based on general population norms. While this provides a non-trivial baseline, it systematically fails to capture the more restrictive behavior of highly privacy-conscious users.
- Experience Trumps Attitude: A user's familiarity with AI ($E_u$) is a more reliable signal for the model than what the user says they care about.
- The "Hardest" Users: The most difficult group to simulate are "low-stance, high-exposure" users—people who use AI a lot but claim not to care about privacy. The LLM often overestimates how much they are willing to share.
Key Results
- Performance Ceiling: The top-performing model, Gemini 3.1 Pro, achieved only 40.4% accuracy when conditioned on experience and attitudes.
- Modest Gains from Personas: Adding persona facets consistently improves simulation quality over a "no-persona" baseline, but the boost is relatively small (roughly 3-4 percentage points).
- Reasoning Doesn't Help (Yet): Interestingly, increasing "reasoning effort" (test-time compute) or using larger model variants within the same family yielded only negligible improvements in accuracy.
Limitations and Future Work
The authors acknowledge that their current persona facets (demographics, experience, attitudes) are likely insufficient for high-fidelity simulation. The "gap between what users say and what they actually do" remains the central obstacle.
Future directions include: * Eliciting richer behavioral signals beyond simple survey responses. * Developing new user studies specifically designed to capture the nuances needed for LLM simulation. * Exploring whether LLMs can be used for low-stakes, early-stage design tasks (like surfacing obvious privacy violations) even if they aren't yet ready to replace human subjects in high-stakes validation.
Summary updated