TL;DR
This paper introduces "Pandora's White-Box," a suite of powerful new attacks that can infer membership of training data in Large Language Models (LLMs) and even extract significant portions of the fine-tuning dataset. These attacks demonstrate a critical privacy vulnerability in LLMs, particularly those that have been fine-tuned.
What Problem Does This Paper Address?
As LLMs become increasingly prevalent and are trained on sensitive data, ensuring their privacy is paramount. Existing privacy attacks, specifically Membership Inference Attacks (MIAs) and training data extraction, have shown limited success against large, pre-trained LLMs, leading some to believe that such models are inherently private. This paper challenges that notion by developing state-of-the-art attacks that significantly outperform previous methods, revealing that LLMs, especially when fine-tuned, are far more vulnerable than previously thought. The authors aim to quantify this risk and highlight the practical implications for LLM security and privacy.
Methods
The paper introduces several novel attack methodologies, categorized by the adversary's access level:
- White-Box MIAs (Pre-trained LLMs): The authors develop supervised MIAs that leverage model gradients. They train classifiers using norms of gradients with respect to input embeddings and various layers of the model. This approach significantly closes the gap between MIA success rates and other machine learning models.
- Gray-Box MIAs (Pre-trained LLMs): By combining model stealing techniques (which can recover model logits from API access) with their supervised attack framework, they create a gray-box MIA that achieves competitive performance without full white-box access.
- Black-Box MIAs (Fine-tuned LLMs): A surprisingly simple attack based on the ratio of loss between the fine-tuned and base models (FLoRa) achieves near-perfect MIA performance on fine-tuned LLMs.
- Data Extraction (Fine-tuned LLMs): Leveraging the strong MIAs, the authors develop a "generate-then-rank" pipeline. By using the FLoRa score as the ranking metric, they can extract a substantial fraction of the fine-tuning dataset.
- Data Extraction (Pre-trained LLMs): While direct MIA-based ranking is less effective for pre-trained models, the authors show that a specifically trained classifier, using similar gradient-norm features, can effectively distinguish generated samples from true training data, enabling extraction.
Datasets
- The Pile: Used for pre-training and fine-tuning evaluation. The authors utilize subsets of The Pile for training and validation.
- Pythia Suite: A range of LLMs (70M to 6.9B parameters) from this suite are used for evaluating pre-trained MIAs.
- Llama-7B and Llama-7B-chat: These models are used for evaluating fine-tuned MIAs and data extraction.
Key Insights and Ideas
- Fine-tuning is a major privacy risk: The paper strongly suggests that fine-tuning LLMs, even for a short period, makes them highly vulnerable to data extraction, with over 50% of the fine-tuning dataset being extractable.
- MIAs are not always good rankers for extraction: An effective MIA doesn't automatically translate to an effective ranking mechanism for data extraction, especially in the pre-trained setting. This necessitates developing specialized classifiers for extraction tasks.
- Gradient norms are powerful features: The norms of model gradients, particularly with respect to input embeddings and layer weights, are crucial features for building effective MIAs, even in a white-box setting.
- Model stealing bridges the gap: The ability to steal model logits from API access significantly weakens the distinction between black-box and white-box attacks, making gray-box attacks a serious concern.
- Small amounts of training data suffice for powerful attacks: Even a tiny fraction of training data (around 1,000 samples) is enough to train highly effective supervised MIAs.
Key Results
- Pre-trained MIAs: New supervised MIAs achieve TPRs hundreds of times better than baselines at low FPRs. For example, on Pythia-1B, their NeuralNet attack achieves a 22% TPR at 0.5% FPR, a 44x increase over random guessing.
- Gray-box MIAs: The ModelStealing attack outperforms previous gray-box MIAs, achieving AUCs up to 0.69.
- Fine-tuned MIAs: The FLoRa attack achieves near-perfect MIA performance on fine-tuned models.
- Fine-tuned Data Extraction: Over 50% of a fine-tuning dataset can be extracted with just 20 generations after 3 epochs of fine-tuning, even with black-box access.
- Pre-trained Data Extraction: A specialized classifier can identify training samples from generations with ~95% accuracy.
Limitations and Future Work
- Computational Cost: While experiments can be run on a single A100 GPU, the compute requirements for training and evaluating these attacks are still significant.
- Practicality of White-Box Access: The most powerful attacks require white-box access, which is not always feasible for proprietary models. The paper explores gray-box alternatives but acknowledges they are slightly less performant.
- Future Work: The authors suggest investigating if supervised white-box attacks can be adapted to practical settings without full model weights or clean train/validation splits. They also propose exploring other techniques for MIAs with high confidence and investigating if FLoRa's efficacy can be replicated in the pre-training setting. A key open question is whether MIAs can match the efficacy of current white-box attacks for models like GPT-4 where full access is unavailable.
Summary updated