TL;DR
This paper argues that current applications of Contextual Integrity (CI) theory to evaluate the privacy implications of Large Language Models (LLMs) are superficial and fail to embrace the theory's core tenets. The authors highlight how this leads to flawed conclusions and designs, urging the ML community to adopt a more rigorous and principled approach to CI for LLMs.
What Problem Does This Paper Address?
The rapid proliferation of Large Language Models (LLMs) has raised significant concerns about their privacy implications. While the academic community has begun to adopt Contextual Integrity (CI) theory as a framework to assess these risks, this paper identifies a critical problem: existing literature inadequately applies CI. Instead of fully embracing its fundamental principles, many researchers use CI in a superficial manner, leading to misinterpretations of privacy violations and the development of ineffective privacy-preserving designs. The authors aim to correct this by clarifying the core tenets of CI and demonstrating how current LLM privacy research deviates from them.
Methods
The paper takes a position-based approach, critically analyzing existing literature that applies CI to LLMs. The authors' "method" involves:
- Clarifying the Four Tenets of CI Theory: They meticulously explain each of the four fundamental tenets of CI, providing a clear understanding of what constitutes a proper CI-based privacy analysis.
- Survey and Systematization of Prior Work: They review nine existing works that have used CI for LLMs (both chatbots and agents). For each work, they systematically assess its alignment (✓) or deviation (✗) from each of the four CI tenets. This creates a structured overview of how CI is being applied (or misapplied).
- Highlighting Experimental Hygiene Issues: They identify crucial experimental considerations for LLM evaluations, specifically prompt sensitivity (how LLM responses vary with minor prompt changes) and position bias (LLMs favoring certain response options), which can lead to unreliable conclusions when not properly accounted for.
Datasets
Not discussed in the paper. The paper focuses on a theoretical critique and systematization of existing research rather than introducing new datasets or benchmarks.
Key Insights and Ideas
- CI is More Than Data Minimization: A significant finding is that many LLM privacy studies conflate CI with simpler notions of privacy like data minimization, public/private dichotomies, or compliance with regulations. True CI is about the appropriateness of information flow within a context, not just restricting certain data types.
- Norms are Contextual, Not Legal: The paper emphasizes that privacy norms are deeply embedded in social contexts and are not equivalent to legal statutes or policies. Relying solely on legal frameworks misses the nuanced, ethical, and societal dimensions that CI aims to capture.
- Crowdsourced Preferences Aren't Norms: Similarly, using user preferences from surveys as a proxy for privacy norms is insufficient. CI recognizes that norms can sometimes conflict with individual preferences for the greater social good.
- The CI Heuristic is Crucial and Overlooked: The CI heuristic (T4) for assessing the ethical legitimacy of norm-breaching information flows is vital for a normative analysis. Its absence in current LLM research means that potentially beneficial but novel information flows might be incorrectly flagged as privacy violations.
- Experimental Rigor is Lacking: The paper highlights that LLMs are highly sensitive to prompt variations and can exhibit position bias. Failing to account for these experimental hygiene issues can lead to unreliable and misleading conclusions about LLM privacy.
Key Results
The paper's "results" are primarily qualitative, stemming from the systematic analysis of prior work. The key findings are:
- Widespread Deviation from CI Tenets: Table 1 and the accompanying text reveal that all surveyed works exhibit significant deviations from at least one, and often multiple, core CI tenets. Most works show deviations across T1, T2, and T4.
- Inadequate Application of CI: The authors conclude that existing LLM literature inadequately applies CI, leading to "flawed privacy-preserving system designs" and "incorrect conclusions."
- Specific Deviations Identified:
- T1 (Appropriate Flow): Often reduced to data minimization or public/private distinctions.
- T2 (Conformity with Norms): Relies on proxies like legal statutes or crowdsourced preferences instead of true contextual norms.
- T3 (Five Parameters): While many works use the parameters, some omit them or provide vague descriptions, hindering comprehensive analysis.
- T4 (CI Heuristic): Almost universally ignored, preventing a normative assessment of ethical legitimacy.
Limitations and Future Work
Limitations:
- Scope of Survey: While nine works are analyzed, the authors acknowledge that this is not an exhaustive list of all CI applications to LLMs.
- Complexity of CI: The paper itself notes that CI is "intuitive but not easy-to-implement," which contributes to the observed superficial applications.
Future Work/Recommendations:
- Embrace Core Tenets: The ML community needs to fully embrace all four core tenets of CI, particularly the normative aspects (T2 and T4).
- Sociotechnical Approach: Applying CI to LLMs requires a sociotechnical perspective, incorporating governance, policy, cultural aspects, and societal values, not just algorithmic solutions.
- Deliberative Process for Norms: Achieving consensus on privacy norms (T2) may require deliberate processes involving diverse expert groups.
- Utilize CI Heuristic: The CI heuristic (T4) should be used to assess the ethical legitimacy of novel or norm-breaching information flows, allowing for potential societal benefits to be recognized.
- Improve Experimental Hygiene: Researchers must account for prompt sensitivity and position bias in their LLM evaluations to ensure robust and reliable conclusions.
- Interdisciplinary Collaboration: The authors call for closer collaboration between the ML community and information/social scientists to better understand the roles LLMs play in shaping and governing information.
Summary updated