TL;DR
This paper tackles the challenge of reconstructing training data for large language models (LLMs) when only the model weights are public. They introduce a novel gradient-based method, SELECT, that effectively identifies useful training data from a large corpus, enabling the training of a new model with performance comparable to the original. This work has significant implications for understanding data privacy and security in the era of open-weight LLMs.
What Problem Does This Paper Address?
Modern LLMs are often released with their weights publicly available ("open-weights") but their training data kept private ("closed-data"). This creates a significant gap: researchers and practitioners can run these powerful models, but they don't know what data shaped their capabilities. This lack of transparency raises concerns about intellectual property, potential biases embedded in the data, and the ability to reproduce or audit model behavior. While previous work has explored data recovery from model weights in computer vision, applying these methods to discrete language data is challenging due to non-differentiability. This paper aims to bridge this gap by developing a method to approximate the training data of LLMs using only their initial and final model weights.
Methods
The core of the paper is the SELECT (SELection of data via gradient-based Checkpoints) algorithm. SELECT leverages the difference between the initial ("base") model weights ($\theta_0$) and the finetuned ("final") model weights ($\theta_f$) to guide data selection. The key idea is that the gradient of the loss function for a data point, when applied to the base model, should ideally point in the direction of $\theta_f - \theta_0$.
Here's a breakdown of the SELECT method:
- Synthetic Checkpoints: Since only two checkpoints ($\theta_0$ and $\theta_f$) are available, the authors create intermediate synthetic checkpoints by linearly interpolating between them. This helps capture a more nuanced "trajectory" of the finetuning process.
- Per-Example Gradients: They compute gradients for each individual data point in a large candidate corpus. To make this computationally feasible, they focus on the last layer's gradients, which are known to be informative and recoverable.
- Low-Dimension Gradient Projection: To handle the high dimensionality of gradients and large corpora, they use the Johnson-Lindenstrauss lemma to project gradients into a lower-dimensional space while preserving inner products. This significantly reduces storage and computation costs.
- Greedy Data Selection: SELECT then employs a greedy approach to select data points. It iteratively picks data points whose projected gradients, when summed, best align with the direction of the model weight difference ($\theta_f - \theta_0$). This submodular optimization problem is solved efficiently.
- Autolabeling: For classification tasks, the final model is used to assign pseudo-labels to the selected data points from the public corpus.
Datasets
The paper uses several datasets for training and evaluation:
- Classification Tasks:
- AG News (4 classes)
- DBPedia (14 classes)
- 20-Newsgroup
- Supervised Finetuning (SFT) Task:
- MSMARCO web documents
- Seed Datasets (for initial unsupervised data):
- Wikipedia passages from Natural Questions
- Other Wikipedia passages for exploring seed data distribution.
The paper does not introduce new benchmark datasets but rather uses existing ones to evaluate their data recovery method.
Key Insights and Ideas
- Gradient Direction as a Proxy for Data Influence: The core insight is that the difference in model weights between an initial and finetuned model encodes information about the data that drove that change. By aligning gradients with this difference, one can identify data points that were influential in the finetuning process.
- Leveraging Discrete Data Challenges: The paper cleverly navigates the non-differentiability of discrete text data by focusing on gradient information and using pseudo-labeling, rather than trying to generate data from scratch.
- Efficiency is Key: The use of last-layer gradients and dimensionality reduction (Johnson-Lindenstrauss) are crucial for making the approach scalable to large corpora.
- Open Weights Reveal More Than Expected: The results suggest that even without direct access to training data, model weights can reveal enough information to approximate effective training subsets.
Key Results
- Classification Performance: On the AG News task, SELECT improved performance from 65% (random selection) to 80%, approaching the expert benchmark of 88%.
- SFT Performance: For models finetuned on MSMARCO, SELECT reduced perplexity from 3.3 to 2.3, compared to an expert LLAMA model's perplexity of 2.0.
- Data Recovery Metrics: Using Optimal Transport (OT) distance and vocabulary containment similarity, SELECT demonstrated superior data recovery compared to baselines like random selection, Top-K, and P-Max.
- Scalability: The method shows promise for scaling to millions of documents, with performance improving with larger projection dimensions for gradients.
- Leakage Analysis: The paper shows that even a small percentage of "leaked" true finetuning data within a larger public dataset significantly improves SELECT's accuracy and reduces OT distance.
Limitations and Future Work
- Threat Model Constraints: The method relies on having access to both the initial and final model weights, as well as knowledge of the optimizer used. This might not always be available.
- Computational Cost: While efficiency improvements are made, computing per-example gradients and performing greedy selection can still be computationally intensive for extremely large datasets.
- "Approximation" vs. "Reconstruction": The method approximates the training data by finding similar data from a public corpus, rather than perfectly reconstructing the original, unknown data.
- Future Work: The authors plan to explore more sophisticated gradient matching techniques and investigate the ethical implications of data recovery more deeply. They also suggest exploring the impact of different optimizers on the effectiveness of their method.
Summary updated