TL;DR
This paper introduces Rényi Differential Privacy (RDP) as a stronger privacy guarantee against reconstruction attacks on language models. They demonstrate that RDP can provide meaningful protection for rare data samples, even when standard differential privacy might fail, and propose an efficient method to identify at-risk training data.
What Problem Does This Paper Address?
When machine learning models, especially large language models (LLMs), are released, there's a risk that adversaries can reconstruct sensitive data samples from the original training set. This is achieved through "reconstruction attacks," where an adversary uses the trained model to regenerate data that was likely part of its training. While differential privacy (DP) is a standard defense, its practical application often uses large privacy budgets (high epsilon values) that offer weak guarantees against these attacks. This paper addresses the gap by showing how Rényi Differential Privacy (RDP) can offer stronger, more nuanced privacy guarantees against reconstruction attacks, particularly for rare or "secret" data.
Methods
The core of the paper's contribution lies in leveraging Rényi Differential Privacy (RDP). RDP is a generalization of standard DP that allows for tighter composition and more precise privacy accounting. The authors derive a new theoretical bound on information leakage from a model trained with RDP. This bound, denoted as $L(p_0)$, quantifies the privacy loss for a secret with prior probability $p_0$. A key insight is that this bound is often tighter than existing DP guarantees, especially for lower levels of noise.
They also introduce a practical method for identifying at-risk training samples. This involves a "lazy sampling" approach to efficiently run reconstruction attacks. Instead of generating vast numbers of samples, they can directly compute the probability of a specific secret being generated by the model. This allows them to pinpoint training data that is particularly vulnerable to extraction.
Datasets
- Wikitext-103: Used for fine-tuning GPT-2 models.
- OpenWebText: Used for training a vanilla language model without DP to identify at-risk samples.
- Canary Data: Custom-generated sequences of digits used for experiments, particularly in the n-gram model and GPT-2 fine-tuning sections.
Key Insights and Ideas
- RDP offers stronger guarantees for reconstruction: Unlike standard DP, RDP can provide meaningful privacy for rare data points (secrets) even with larger privacy budgets. This means that while common data might be more exposed, unique or sensitive information remains better protected.
- The length of a secret matters for privacy: Longer secrets are inherently harder to reconstruct, and RDP's guarantees reflect this. The paper shows that the length of the secret itself acts as a form of privacy protection.
- Lazy sampling is efficient for reconstruction attacks: This technique significantly speeds up the process of identifying vulnerable training samples, making practical analysis of reconstruction risks more feasible.
- Membership inference vs. reconstruction: The paper highlights that privacy settings that might not fully prevent membership inference (identifying if a specific sample was in the training set) can still effectively prevent the full reconstruction of rare secrets.
Key Results
- The proposed RDP-based leakage bound is shown to be tighter than existing DP bounds, particularly for lower noise levels.
- Experiments with an n-gram language model demonstrate that the empirical leakage closely matches the theoretical bound.
- When applying their lazy sampling attack to a non-private language model trained on OpenWebText, they successfully identify training samples at high risk of reconstruction.
- Fine-tuning a GPT-2 model on Wikitext-103 with DP-SGD achieved competitive perplexity while exhibiting empirical leakage that was even lower than their theoretical guarantee.
Limitations and Future Work
- "Canary" vs. Real-world secrets: The paper uses "canary" sentences (randomly generated digits) as a proxy for secrets. While useful for analysis, real-world secrets might have inherent structure or prior probabilities that differ from these artificial canaries. The authors acknowledge that practical reconstruction attacks face challenges in estimating the randomness of secrets and accounting for knowledge shared across similar secrets.
- Attack optimality: The paper's analysis relies on the assumption that the attack mechanisms are reasonably effective. However, they note that they can only "upper-bound the exposure because we have no guarantee that any attack is optimal."
- Future work: The authors suggest that their work sheds light on the "higher information" end of the privacy spectrum and hopes that increased consideration for this threat model will drive the adoption of DP-SGD as a standard in machine learning. They also implicitly point towards further research into more sophisticated reconstruction attacks and defenses.
Summary updated