Introduction
As Large Language Models (LLMs) become central to digital life, they are increasingly trained on massive, uncurated datasets from the open web, such as Common Crawl or The Pile. These datasets often contain sensitive Personally Identifiable Information (PII)—including names, home addresses, and social security numbers—that was never intended for public AI training. The core challenge is that LLMs do not just learn language; they often "memorize" specific data points, creating a significant privacy surface area where an adversary might elicit a private individual's contact information.
This systematic survey provides a comprehensive map of how LLMs inadvertently leak PII, categorizing the evolving landscape of extraction attacks and evaluating current defense strategies. It systematizes the research into three main pillars: attack taxonomy, defense taxonomy, and evaluation frameworks.
Attack Taxonomy
The research categorizes PII extraction into three primary methodologies based on the attack vector:
- Extraction from Leaked Training Data: These attacks utilize "divergence" techniques, inducing the model to produce repetitive or nonsensical outputs that eventually reveal verbatim snippets from its training corpus.
- Crafted Prompts: Attackers use sophisticated prompt engineering, such as Chain-of-Thought (CoT) reasoning, few-shot examples, or "true-prefixes" (providing a known portion of a target's info), to trigger the model to complete the sensitive record.
- Fine-tuning Exploits: Demonstrating how the fine-tuning process can inadvertently "restore" or "unlock" PII that appeared to have been suppressed during the model's initial pre-training or alignment.
Defense Taxonomy
Defenses are categorized based on the stage of the model lifecycle they address:
- Training Stage: Focuses on rigorous data cleaning and the implementation of Differential Privacy (DP) during pre-training.
- Model Adaptation: Includes "unlearning" specific data points or identifying and deactivating "privacy neurons"—specific weights in the Transformer layers (often the upper MLP layers) that are responsible for storing sensitive information. Deactivating these neurons can significantly drop leakage (e.g., from 45% to 5%) without destroying general intelligence.
- Query Execution: Employs real-time filtering, such as the PAPILLON framework, which uses a local, smaller LLM to sanitize prompts and outputs before they reach a more powerful remote model.
Datasets and Evaluation
The field relies heavily on a few key datasets, most notably the Enron Email Dataset, though its dominance has led to concerns about "Enron bias." Because Enron represents a specific professional era, current state-of-the-art results might not generalize to modern PII. Other critical sources include The Pile, Common Crawl, and the specialized Ai4Privacy dataset. Emerging benchmarks like PII-Scope and PII-Compass are beginning to standardize how model vulnerability is assessed using metrics like Attack Success Rate (ASR) and Hit@n.
Key Insights and Results
Research indicates that PII leakage is not merely a matter of memorization but also of association. Models can often reconstruct full email addresses if they have associated a name with a specific organization during training.
Empirical results highlight the severity of the issue:
- Extraction Efficacy: Studies on GPT-4 and GPT-3.5 showed PII extraction rates of 48.19% and 44.47% respectively when targeting Enron email addresses.
- Divergence Attacks: Researchers found that "divergence attacks" on Llama-2 could result in 16.9% of responses containing authentic memorized PII.
- Defense Success: The PAPILLON hybrid defense was shown to reduce PII leakage from 100% down to 7.5% with minimal impact on response quality. Techniques like REVS (Rank Editing in the Vocabulary Space) achieved nearly 99% deletion accuracy for Social Security Numbers.
Limitations and Future Directions
Current research is largely focused on "exact match" extraction and targeted attacks. Future work must address semantic leakage, where private information is revealed indirectly. Furthermore, there is an urgent need to diversify datasets beyond the Enron corpus to reflect modern data types and to develop proactive defenses that can detect malicious intent within prompts before a response is generated.
Summary updated