TL;DR
This paper introduces PII-Compass, a novel method to improve the extraction of Personally Identifiable Information (PII) from Large Language Models (LLMs). By grounding manually crafted prompts with true prefixes from related data, PII-Compass significantly boosts extraction rates, demonstrating a practical approach to assessing and mitigating privacy risks in LLMs.
What Problem Does This Paper Address?
Large Language Models (LLMs) have become incredibly powerful, but their impressive memorization capabilities raise significant privacy concerns. Specifically, LLMs can sometimes regurgitate sensitive Personally Identifiable Information (PII) that was present in their training data. While methods exist to detect and quantify this risk, there's no consensus on optimal evaluation strategies, potentially leading to an underestimation of real-world adversarial capabilities. This paper focuses on the challenging scenario of "black-box" LLM access, where an attacker can only query the model without direct access to its internal workings or training data. The core problem is how to effectively extract PII from such models, especially when direct knowledge of the training data's structure is unavailable.
Methods
The paper's central contribution is the PII-Compass method, which aims to improve PII extraction by "grounding" prompts. The intuition is that a prompt's embedding should be close to the embedding of the target PII and its associated context for successful extraction.
- Baseline: The authors first evaluate simple, hand-crafted prompt templates (e.g., "What is the phone number of {name}?"). They find these are largely ineffective, achieving extraction rates below 1%.
- True-Prefix Prompting: They then explore using actual prefixes from the training data associated with the target PII. This significantly improves extraction rates, showing that longer prefixes lead to better results, up to a point.
- PII-Compass: The novel approach involves prepending a manually crafted prompt template with a true prefix from a different data subject (but from the same domain) found in an "adversary dataset." This "grounding" prompt, while not an exact match to the target's context, nudges the LLM's internal representations closer to where PII is likely to be found, thereby increasing the probability of extraction. They also explore using prefixes from different domains (out-domain) to show the importance of domain relevance.
Datasets
- Enron Email Dataset: A post-processed version of the Enron email dataset was used. This dataset was filtered to contain pairs of names and phone numbers, resulting in 2,080 data subjects.
- PILE Corpus: The Enron dataset is part of the larger PILE corpus, which was used for training the target LLM.
- Adversary Dataset: A subset of 128 data subjects from the Enron dataset was designated as the "Adversary dataset," from which true prefixes were drawn.
- Evaluation Dataset: The remaining 1,952 data subjects formed the "Evaluation dataset" for testing extraction performance.
Key Insights and Ideas
- Manual prompts are weak: Simple, hand-crafted prompts are surprisingly ineffective for PII extraction, highlighting the need for more sophisticated methods.
- Prefixes matter, but context is key: While using true prefixes of the target PII is effective, PII-Compass shows that using prefixes from related but different data subjects can achieve similar or even better results without needing direct access to the target's exact prefix.
- Embedding space matters: The paper visualizes prompt embeddings and shows that PII-Compass successfully moves the embeddings of manual prompts closer to the embeddings of true prefixes, explaining its effectiveness.
- Domain grounding is crucial: Using out-domain prefixes (e.g., from GitHub code examples) did not improve extraction and even pushed embeddings away from the target region, underscoring the importance of domain-relevant grounding.
Key Results
- Baseline (Manual Templates): PII extraction rates were less than 0.15% for all tested templates.
- True-Prefix Prompting: Extraction rates increased significantly with prefix length, reaching up to 5.94% with 100-token prefixes.
- PII-Compass:
- Achieved extraction rates of 0.92% with 1 query, 3.9% with 128 queries, and 6.86% with 2308 queries.
- This represents a 5-18 times higher extraction rate compared to baseline manual templates.
- The method allows for extracting the phone number of 1 in 15 people with a reasonable query budget.
- The aggregated PII extraction rate (at least one success in 128 queries across templates) reached 3.89%.
Limitations and Future Work
- Limited PII types: The study was limited to phone numbers due to the lack of publicly available datasets for other PII types like credit card numbers or SSNs.
- Adversary dataset assumption: The method assumes access to true prefixes from an adversary dataset.
- Annotation source: PII annotations were extracted using GPT-4 and then pruned, with manual verification on a subset. The authors acknowledge potential annotation errors.
- Future Work: The authors suggest exploring the use of zero-shot capabilities of GPT-4 to generate prefixes that could guide extraction even without an adversary dataset. They also aim to study extraction rates for other PII types.
Summary updated