"It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents

Zhiping Zhang, Michelle Jia, Hao-Ping Lee, Bingsheng Yao, Sauvik Das, Ada Lerner, Dakuo Wang, Tianshi Li

TL;DR

This paper investigates how everyday users navigate privacy trade-offs when sharing sensitive information with LLM-based conversational agents like ChatGPT. Through a qualitative analysis of real-world chat logs and interviews with 19 users, the authors reveal that human-like interactions nudge users into disclosing deeply personal data, while flawed mental models and interface dark patterns severely hinder their ability to protect themselves.

What Problem Does This Paper Address?

Large Language Model (LLM)-based conversational agents (CAs) are rapidly expanding into high-stakes domains like healthcare, finance, and legal counseling. To leverage these tools effectively, users frequently input highly sensitive data, such as medical records, financial statements, or proprietary code.

While substantial technical research focuses on model-centric privacy (e.g., measuring and mitigating data memorization at the algorithmic level), there is a significant gap in understanding the human side of the equation. This paper addresses this gap by exploring users' actual disclosure behaviors, their motivations, their cognitive understanding of how LLMs handle data, and the systemic design barriers that prevent them from effectively managing their privacy.

Methods

The authors employed a two-pronged, human-centered empirical approach:

  1. In-situ Dataset Analysis: To observe real-world behavior, the authors analyzed a sample of public ChatGPT conversation histories. Using automated Personally Identifiable Information (PII) detection followed by manual qualitative coding grounded in the Contextual Integrity framework, they analyzed 200 conversation threads (comprising 10,380 messages) to build a multidimensional typology of sensitive disclosures.
  2. Semi-Structured Interviews: The authors conducted interviews with 19 regular users of LLM CAs. A core component of the interview was a "mental model drawing activity" where participants sketched out their understanding of how data flows through the system behind the scenes. This was followed by a debriefing session evaluating the users' awareness of actual LLM training mechanisms and existing privacy controls.

Datasets

  • ShareGPT52K: A public dataset containing 50,496 ChatGPT chat histories shared by users via a Chrome extension between December 2022 and March 2023. The authors filtered this down to 30,000 threads containing PII, oversampled dense PII threads, and qualitatively analyzed a final saturated sample of 200 conversation threads.
  • No new dataset benchmarks were introduced; the dataset was used purely for empirical behavior modeling.

Key Insights and Ideas

  • The Anthropomorphism Nudge: The open-ended, human-like nature of LLMs acts as a psychological nudge. Users frequently treat the CA as a "friend" or "therapist," leading to a conversational flow where they gradually reveal highly sensitive details over multiple rounds—sometimes explicitly prompted by the AI's empathetic framing.
  • Interdependent Privacy Risks: Users do not just share their own data; they routinely paste emails, code snippets, or documents containing the PII of colleagues, clients, and friends without their explicit consent, complicating data ownership boundaries.
  • Broken Mental Models: Users hold highly inaccurate cognitive maps of LLMs. Many view the system as a "super searcher" (like a live Google search) or believe their inputs act purely as an isolated "quality indicator" (similar to a thumbs up/down feedback loop). They fundamentally fail to realize that their inputs can be absorbed into a global training set and potentially leaked to other users via model memorization.
  • The Illusion of a "Fair Game": While users often rationalize the privacy-for-utility trade-off as a fair transaction, interface design choices make it structurally unfair by obscuring risks and complicating user control.

Key Results

  • Typology of Disclosure: The authors mapped out sensitive disclosures across various topics (Business, Legal, Medical, Financial, etc.), highlighting how users share everything from crypto transaction histories to raw medical codes.
Figure 1: A fictional example of sensitive disclosure to ChatGPT inspired by real-world use cases: A user shared their doctor’s email and ICD-10-CM diagnosis results with ChatGPT upon its request. And then ChatGPT interpreted the codes, indicating the user had multiple diseases. Three issues are demonstrated in the example: 1. disclosed PII (name) and non-identifiable but sensitive information (diagnosis results); 2. disclosed other person’s information (doctor’s information); 3. ChatGPT actively requested for detailed information from the user which encouraged user’s disclosure behavior.
  • Dark Patterns Exposed: A major finding was that ChatGPT's privacy controls employ dark patterns. The easiest way to opt out of model training forces users to turn off their chat history entirely. The alternative—a standalone opt-out form that preserves history—was hidden deep within an external FAQ document. Not a single interview participant found this form independently.
  • Ad-Hoc Sanitization: To cope with privacy fears, users resort to tedious, manual workarounds like falsifying names, using generic placeholders, or segmenting text into tiny chunks to prevent the model from capturing the full context of an unpublished paper.

Limitations and Future Work

  • Platform Bias: The study heavily focused on ChatGPT, meaning the findings might not perfectly translate to other CAs with different interaction paradigms or user demographics.
  • Sample Demographics: The interview sample consisted of general population users; the authors note that vulnerable groups (e.g., older adults, non-native speakers, or dedicated therapy seekers) might face higher opportunity costs and different privacy dynamics that warrant dedicated study.
  • Need for Quantitative Modeling: Future work should quantitatively model the impact of these mental models and contextual factors on user behavior to guide system design.
  • Longitudinal Tracking: As LLM technology becomes mainstream and public awareness shifts, longitudinal studies will be essential to track how user mental models and privacy attitudes evolve over time.

Summary updated

Abstract

The widespread use of Large Language Model (LLM)-based conversational agents (CAs), especially in high-stakes domains, raises many privacy concerns. Building ethical LLM-based CAs that respect user privacy requires an in-depth understanding of the privacy risks that concern users the most. However, existing research, primarily model-centered, does not provide insight into users' perspectives. To bridge this gap, we analyzed sensitive disclosures in real-world ChatGPT conversations and conducted semi-structured interviews with 19 LLM-based CA users. We found that users are constantly faced with trade-offs between privacy, utility, and convenience when using LLM-based CAs. However, users' erroneous mental models and the dark patterns in system design limited their awareness and comprehension of the privacy risks. Additionally, the human-like interactions encouraged more sensitive disclosures, which complicated users' ability to navigate the trade-offs. We discuss practical design guidelines and the needs for paradigm shifts to protect the privacy of LLM-based CA users.
BibTeX
@inproceedings{zhang2023it,
  author = {Zhiping Zhang and Michelle Jia and Hao-Ping Lee and Bingsheng Yao and Sauvik Das and Ada Lerner and Dakuo Wang and Tianshi Li},
  title = {{"It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents}},
  booktitle = {CHI},
  year = {2023},
}
Figure