TL;DR
This paper investigates how everyday users navigate privacy trade-offs when sharing sensitive information with LLM-based conversational agents like ChatGPT. Through a qualitative analysis of real-world chat logs and interviews with 19 users, the authors reveal that human-like interactions nudge users into disclosing deeply personal data, while flawed mental models and interface dark patterns severely hinder their ability to protect themselves.
What Problem Does This Paper Address?
Large Language Model (LLM)-based conversational agents (CAs) are rapidly expanding into high-stakes domains like healthcare, finance, and legal counseling. To leverage these tools effectively, users frequently input highly sensitive data, such as medical records, financial statements, or proprietary code.
While substantial technical research focuses on model-centric privacy (e.g., measuring and mitigating data memorization at the algorithmic level), there is a significant gap in understanding the human side of the equation. This paper addresses this gap by exploring users' actual disclosure behaviors, their motivations, their cognitive understanding of how LLMs handle data, and the systemic design barriers that prevent them from effectively managing their privacy.
Methods
The authors employed a two-pronged, human-centered empirical approach:
- In-situ Dataset Analysis: To observe real-world behavior, the authors analyzed a sample of public ChatGPT conversation histories. Using automated Personally Identifiable Information (PII) detection followed by manual qualitative coding grounded in the Contextual Integrity framework, they analyzed 200 conversation threads (comprising 10,380 messages) to build a multidimensional typology of sensitive disclosures.
- Semi-Structured Interviews: The authors conducted interviews with 19 regular users of LLM CAs. A core component of the interview was a "mental model drawing activity" where participants sketched out their understanding of how data flows through the system behind the scenes. This was followed by a debriefing session evaluating the users' awareness of actual LLM training mechanisms and existing privacy controls.
Datasets
- ShareGPT52K: A public dataset containing 50,496 ChatGPT chat histories shared by users via a Chrome extension between December 2022 and March 2023. The authors filtered this down to 30,000 threads containing PII, oversampled dense PII threads, and qualitatively analyzed a final saturated sample of 200 conversation threads.
- No new dataset benchmarks were introduced; the dataset was used purely for empirical behavior modeling.
Key Insights and Ideas
- The Anthropomorphism Nudge: The open-ended, human-like nature of LLMs acts as a psychological nudge. Users frequently treat the CA as a "friend" or "therapist," leading to a conversational flow where they gradually reveal highly sensitive details over multiple rounds—sometimes explicitly prompted by the AI's empathetic framing.
- Interdependent Privacy Risks: Users do not just share their own data; they routinely paste emails, code snippets, or documents containing the PII of colleagues, clients, and friends without their explicit consent, complicating data ownership boundaries.
- Broken Mental Models: Users hold highly inaccurate cognitive maps of LLMs. Many view the system as a "super searcher" (like a live Google search) or believe their inputs act purely as an isolated "quality indicator" (similar to a thumbs up/down feedback loop). They fundamentally fail to realize that their inputs can be absorbed into a global training set and potentially leaked to other users via model memorization.
- The Illusion of a "Fair Game": While users often rationalize the privacy-for-utility trade-off as a fair transaction, interface design choices make it structurally unfair by obscuring risks and complicating user control.
Key Results
- Typology of Disclosure: The authors mapped out sensitive disclosures across various topics (Business, Legal, Medical, Financial, etc.), highlighting how users share everything from crypto transaction histories to raw medical codes.
- Dark Patterns Exposed: A major finding was that ChatGPT's privacy controls employ dark patterns. The easiest way to opt out of model training forces users to turn off their chat history entirely. The alternative—a standalone opt-out form that preserves history—was hidden deep within an external FAQ document. Not a single interview participant found this form independently.
- Ad-Hoc Sanitization: To cope with privacy fears, users resort to tedious, manual workarounds like falsifying names, using generic placeholders, or segmenting text into tiny chunks to prevent the model from capturing the full context of an unpublished paper.
Limitations and Future Work
- Platform Bias: The study heavily focused on ChatGPT, meaning the findings might not perfectly translate to other CAs with different interaction paradigms or user demographics.
- Sample Demographics: The interview sample consisted of general population users; the authors note that vulnerable groups (e.g., older adults, non-native speakers, or dedicated therapy seekers) might face higher opportunity costs and different privacy dynamics that warrant dedicated study.
- Need for Quantitative Modeling: Future work should quantitatively model the impact of these mental models and contextual factors on user behavior to guide system design.
- Longitudinal Tracking: As LLM technology becomes mainstream and public awareness shifts, longitudinal studies will be essential to track how user mental models and privacy attitudes evolve over time.
Summary updated