CLIOPATRA: Extracting Private Information from LLM Insights

Meenatchi Sundaram Muthu Selva Annamalai, Emiliano De Cristofaro, Peter Kairouz

TL;DR

Researchers have unveiled Cliopatra, the first targeted privacy attack against LLM-based insight systems like Anthropic’s Clio. By injecting a small number of "poisoned" chats, an adversary can bypass multiple layers of heuristic defenses—including PII redaction and LLM-based auditing—to extract sensitive medical information from a target user with up to 100% success in certain configurations.

What Problem Does This Paper Address?

As AI assistants become ubiquitous, companies want to analyze user interactions to improve their services. However, these chats often contain highly sensitive personal or corporate data. To address this, platforms like Clio were developed to provide "privacy-preserving" insights.

Clio relies on a "defense-in-depth" strategy using four heuristic stages:

  1. Facet Extraction: Summarizing the "user request" while redacting Personally Identifiable Information (PII).
  2. Clustering: Grouping similar requests together to hide individual users in a crowd.
  3. Summarization: Generating a high-level overview of each cluster.
  4. Privacy Auditing: Using an LLM to "grade" the final summary for privacy leaks.

The authors of this paper challenge the assumption that these layered heuristics provide robust protection, demonstrating that without formal guarantees (like Differential Privacy), these systems remain vulnerable to sophisticated poisoning attacks.

Methods

The authors introduce Cliopatra, a targeted poisoning attack designed to "break" the Clio pipeline. The attack works in three main phases:

  • Crafting the Poison: The adversary creates "poison chats" using a three-stage prompt injection. They use a trigger phrase to ensure the poison chats cluster with the target user's chat, a summarization injection to trick the final LLM into revealing the target's private data, and an extractor wrapper to ensure the malicious instructions survive the initial PII redaction phase.
  • Injection: The adversary inserts $C-1$ copies of this poison chat (where $C$ is the minimum cluster size) into the system via fake accounts.
  • Extraction: Once Clio processes the data, the adversary inspects the public cluster summaries. They use either regex matching or a powerful LLM (like Claude 3.5 Sonnet) to identify the poisoned cluster and extract the target's private information (e.g., a specific medical diagnosis).

The authors also developed a "Stealthy Cliopatra" variant that replaces overt identifiers (like age/gender) with random numerical IDs, making the attack harder for human inspectors or automated filters to spot.

Datasets

  • WildChat: A dataset of 1M real-world user-AI interactions. The authors used random samples of 1K to 100K chats as "background noise."
  • NLICE: A synthetic medical record database used to create realistic "personas" (age, gender, symptoms, and diseases).
  • Synthetic Target Chats: The authors used Claude 3.5 Sonnet to generate realistic medical conversations based on the NLICE personas to serve as the targets for extraction.

Key Insights and Ideas

  • Heuristics are Not a Fortress: The "defense-in-depth" approach fails because each layer is susceptible to prompt injection. If the first layer (facet extraction) is compromised, the subsequent layers are processing "poisoned" data.
  • LLMs are Poor Privacy Auditors: One of the most striking findings is that using an LLM to audit another LLM's output for privacy leaks is highly unreliable. The "auditor" LLM frequently gave high privacy scores (5/5) to summaries that clearly leaked medical diagnoses, often justifying the score by claiming the information was "generic."
  • Information Leaks Early: Even without an attack, the initial "facet extraction" stage often fails to redact PII. Attributes like age and gender—which can be used for re-identification—leaked in up to 63% of cases across various model families.
  • Confidence is Key: The attack is so effective that an adversary can often tell exactly when they have succeeded, allowing them to ignore "failed" attempts and maintain high precision.

Key Results

  • High Success Rates: Against the standard Clio configuration (using Claude models), Cliopatra extracted the correct medical diagnosis in 39% of cases with only one known symptom.
  • Model Sensitivity: The attack's effectiveness varied by model. When Clio was configured with Qwen 3, success jumped to 81%.
  • Knowledge Scaling: With more information (e.g., knowing 5 of the target's symptoms), the attack success reached nearly 100% for some model configurations.
  • Scale Vulnerability: While increasing the total number of chats (e.g., to 100K) makes clustering harder, the attack remained successful for a significant number of users, proving that "hiding in a crowd" isn't a guarantee of safety against a motivated attacker.
  • Mitigation: The authors tested URANIA, a system using formal Differential Privacy (DP). They found that DP provided much stronger protection than heuristics, though it comes with a trade-off in the "utility" or detail of the insights generated.

Limitations and Future Work

  • Synthetic Focus: The evaluation relied on synthetic medical chats. While these were designed to be realistic, real-world conversations might be more complex or fragmented.
  • Single-Chat Targeting: The current attack targets one user chat at a time. Future work could explore "untargeted" attacks aimed at exfiltrating as much data as possible from a wide range of users simultaneously.
  • Multimodal Data: As AI assistants move toward processing images and files, the authors suggest investigating how Cliopatra-style attacks might evolve in multimodal insight systems.

Summary updated

Abstract

As AI assistants become widely used, privacy-aware platforms like Anthropic's Clio have been introduced to generate insights from real-world AI use. Clio's privacy protections rely on layering multiple heuristic techniques together, including PII redaction, clustering, filtering, and LLM-based privacy auditing. In this paper, we put these claims to the test by presenting CLIOPATRA, the first privacy attack against "privacy-preserving" LLM insight systems. The attack involves a realistic adversary that carefully designs and inserts malicious chats into the system to break multiple layers of privacy protections and induce the leakage of sensitive information from a target user's chat. We evaluated CLIOPATRA on synthetically generated medical target chats, demonstrating that an adversary who knows only the basic demographics of a target user and a single symptom can successfully extract the user's medical history in 39% of cases by just inspecting Clio's output. Furthermore, CLIOPATRA can reach close to 100% when Clio is configured with other state-of-the-art models and the adversary's knowledge of the target user is increased. We also show that existing ad hoc mitigations, such as LLM-based privacy auditing, are unreliable and fail to detect major leaks. Our findings indicate that even when layered, current heuristic protections are insufficient to adequately protect user data in LLM-based analysis systems.
BibTeX
@misc{annamalai2026cliopatra,
  author = {Meenatchi Sundaram Muthu Selva Annamalai and Emiliano De Cristofaro and Peter Kairouz},
  title = {{CLIOPATRA: Extracting Private Information from LLM Insights}},
  year = {2026},
  eprint = {2603.09781},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.09781},
}
Figure