TL;DR
Researchers have unveiled Cliopatra, the first targeted privacy attack against LLM-based insight systems like Anthropic’s Clio. By injecting a small number of "poisoned" chats, an adversary can bypass multiple layers of heuristic defenses—including PII redaction and LLM-based auditing—to extract sensitive medical information from a target user with up to 100% success in certain configurations.
What Problem Does This Paper Address?
As AI assistants become ubiquitous, companies want to analyze user interactions to improve their services. However, these chats often contain highly sensitive personal or corporate data. To address this, platforms like Clio were developed to provide "privacy-preserving" insights.
Clio relies on a "defense-in-depth" strategy using four heuristic stages:
- Facet Extraction: Summarizing the "user request" while redacting Personally Identifiable Information (PII).
- Clustering: Grouping similar requests together to hide individual users in a crowd.
- Summarization: Generating a high-level overview of each cluster.
- Privacy Auditing: Using an LLM to "grade" the final summary for privacy leaks.
The authors of this paper challenge the assumption that these layered heuristics provide robust protection, demonstrating that without formal guarantees (like Differential Privacy), these systems remain vulnerable to sophisticated poisoning attacks.
Methods
The authors introduce Cliopatra, a targeted poisoning attack designed to "break" the Clio pipeline. The attack works in three main phases:
- Crafting the Poison: The adversary creates "poison chats" using a three-stage prompt injection. They use a trigger phrase to ensure the poison chats cluster with the target user's chat, a summarization injection to trick the final LLM into revealing the target's private data, and an extractor wrapper to ensure the malicious instructions survive the initial PII redaction phase.
- Injection: The adversary inserts $C-1$ copies of this poison chat (where $C$ is the minimum cluster size) into the system via fake accounts.
- Extraction: Once Clio processes the data, the adversary inspects the public cluster summaries. They use either regex matching or a powerful LLM (like Claude 3.5 Sonnet) to identify the poisoned cluster and extract the target's private information (e.g., a specific medical diagnosis).
The authors also developed a "Stealthy Cliopatra" variant that replaces overt identifiers (like age/gender) with random numerical IDs, making the attack harder for human inspectors or automated filters to spot.
Datasets
- WildChat: A dataset of 1M real-world user-AI interactions. The authors used random samples of 1K to 100K chats as "background noise."
- NLICE: A synthetic medical record database used to create realistic "personas" (age, gender, symptoms, and diseases).
- Synthetic Target Chats: The authors used Claude 3.5 Sonnet to generate realistic medical conversations based on the NLICE personas to serve as the targets for extraction.
Key Insights and Ideas
- Heuristics are Not a Fortress: The "defense-in-depth" approach fails because each layer is susceptible to prompt injection. If the first layer (facet extraction) is compromised, the subsequent layers are processing "poisoned" data.
- LLMs are Poor Privacy Auditors: One of the most striking findings is that using an LLM to audit another LLM's output for privacy leaks is highly unreliable. The "auditor" LLM frequently gave high privacy scores (5/5) to summaries that clearly leaked medical diagnoses, often justifying the score by claiming the information was "generic."
- Information Leaks Early: Even without an attack, the initial "facet extraction" stage often fails to redact PII. Attributes like age and gender—which can be used for re-identification—leaked in up to 63% of cases across various model families.
- Confidence is Key: The attack is so effective that an adversary can often tell exactly when they have succeeded, allowing them to ignore "failed" attempts and maintain high precision.
Key Results
- High Success Rates: Against the standard Clio configuration (using Claude models), Cliopatra extracted the correct medical diagnosis in 39% of cases with only one known symptom.
- Model Sensitivity: The attack's effectiveness varied by model. When Clio was configured with Qwen 3, success jumped to 81%.
- Knowledge Scaling: With more information (e.g., knowing 5 of the target's symptoms), the attack success reached nearly 100% for some model configurations.
- Scale Vulnerability: While increasing the total number of chats (e.g., to 100K) makes clustering harder, the attack remained successful for a significant number of users, proving that "hiding in a crowd" isn't a guarantee of safety against a motivated attacker.
- Mitigation: The authors tested URANIA, a system using formal Differential Privacy (DP). They found that DP provided much stronger protection than heuristics, though it comes with a trade-off in the "utility" or detail of the insights generated.
Limitations and Future Work
- Synthetic Focus: The evaluation relied on synthetic medical chats. While these were designed to be realistic, real-world conversations might be more complex or fragmented.
- Single-Chat Targeting: The current attack targets one user chat at a time. Future work could explore "untargeted" attacks aimed at exfiltrating as much data as possible from a wide range of users simultaneously.
- Multimodal Data: As AI assistants move toward processing images and files, the authors suggest investigating how Cliopatra-style attacks might evolve in multimodal insight systems.
Summary updated