Approximating Language Model Training Data from Weights

John X. Morris, Junjie Oscar Yin, Woojeong Kim, Vitaly Shmatikov, Alexander M. Rush

Introduction

In the current AI landscape, the release of "open-weights" models—such as LLaMA-3 or DeepSeek-R1—has become a standard for promoting research. However, while the model parameters are public, the specific datasets used to fine-tune these models are often kept secret. This creates a significant transparency gap: we have access to the final "brain," but the specific information it was "fed" remains unknown. Researchers from Cornell have developed SELECT, a method that bridges this gap by approximating secret fine-tuning datasets using only the model's weights and a large public corpus.

The Problem of Data Reconstruction

Reconstructing training data from model weights is a known challenge. In computer vision, models can often be "inverted" to generate images they were trained on. However, language is discrete, composed of tokens rather than continuous pixels, which causes traditional gradient-based reconstruction methods to fail.

Instead of generating data from scratch, the authors propose a selection strategy. If the training data exists somewhere in the massive "haystack" of the public internet, can we find it—or a mathematically equivalent proxy—by analyzing how the model's weights changed during training?

The SELECT Framework

The authors introduce SELECT (Greedy Dataset Creation with Autolabeling), a framework that operates under a threat model where an adversary has access to the base model weights ($\theta_0$), the fine-tuned weights ($\theta_f$), and a large "seed" corpus of public text (e.g., Wikipedia).

Overview of the SELECT framework, illustrating the four-step process from autolabeling to greedy data selection.

The method follows four primary steps:

  1. Autolabeling: The adversary uses the fine-tuned model itself to generate "pseudo-labels" for the public text in the seed corpus.
  2. Synthetic Trajectories: Since intermediate training checkpoints are usually unavailable, SELECT creates "synthetic checkpoints" by linearly interpolating between the base and final weights to simulate the training path.
  3. Gradient Matching: The core hypothesis is that the training data pushed the model from $\theta_0$ toward $\theta_f$. The algorithm calculates gradients for every document in the seed corpus and identifies those whose gradients align most closely with the direction of the weight change.
  4. Efficiency Optimizations: To handle millions of documents, the authors use Johnson-Lindenstrauss (JL) random projections to compress high-dimensional gradients and focus on the gradients of the model's final layer.

Experimental Evaluation

The researchers evaluated SELECT across classification tasks (AG News, DBPedia, IMDB) and Supervised Fine-Tuning (SFT) tasks using MSMARCO. They primarily used Wikipedia passages as the "haystack" from which to select data.

Key Results

The results demonstrate that SELECT can effectively "steal" the performance of a secret dataset:

  • Closing the Performance Gap: On the AG News task, a model trained on randomly selected data achieved only 65% accuracy. A model trained on data chosen by SELECT reached 80% accuracy, nearing the original model's benchmark of 88%.
  • SFT Perplexity: When approximating a model fine-tuned on MSMARCO, SELECT reduced the model's perplexity from 3.3 to 2.3, approaching the expert LLaMA model's perplexity of 2.0.
  • Robustness: The method remained effective even when the seed data distribution differed from the original training data (e.g., using Wikipedia to approximate news).
Model accuracy and Optimal Transport (OT) distance as a function of the percentage of leaked training data.

Discussion and Insights

The paper argues that for discrete data like text, selection is a more powerful primitive than generation. The difference vector ($\theta_f - \theta_0$) acts as a mathematical "fingerprint" of the training data. Even if the exact training samples are not in the seed corpus, SELECT finds "proxy" data that exerts the same influence on the model.

Furthermore, the method uses greedy submodular optimization to ensure the selected batch is diverse, preventing the redundancy that would occur if one simply picked the top-$k$ most similar documents. This approach is particularly effective at detecting data leakage, raising significant privacy and intellectual property concerns for model creators.

Limitations and Conclusion

Despite its success, SELECT requires the exact base model used for fine-tuning; without it, the gradient signal becomes noisy. Additionally, standard optimization techniques like weight decay can "blur" the weight-level signal, making recovery more difficult. While this work focuses on fine-tuning, reconstructing the massive datasets used for initial pre-training remains an open challenge.

The authors conclude that releasing model weights is not a "zero-risk" move. As approximation techniques improve, the distinction between "open weights" and "open data" continues to blur, necessitating a re-evaluation of privacy in the age of open-source AI.

Summary updated

Abstract

Modern language models often have open weights but closed training data. We formalize the problem of data approximation from model weights and propose several baselines and metrics. We develop a gradient-based approach that selects the highest-matching data from a large public text corpus and show its effectiveness at recovering useful data given only weights of the original and finetuned models. Even when none of the true training data is known, our method is able to locate a small subset of public Web documents can be used to train a model to close to the original model performance given models trained for both classification and supervised-finetuning. On the AG News classification task, our method improves performance from 65% (using randomly selected data) to 80%, approaching the expert benchmark of 88%. When applied to a model trained with SFT on MSMARCO web documents, our method reduces perplexity from 3.3 to 2.3, compared to an expert LLAMA model's perplexity of 2.0.
BibTeX
@inproceedings{morris2025approximating,
  author = {John X. Morris and Junjie Oscar Yin and Woojeong Kim and Vitaly Shmatikov and Alexander M. Rush},
  title = {{Approximating Language Model Training Data from Weights}},
  booktitle = {COLM},
  year = {2025},
  url = {https://openreview.net/forum?id=cQechnXCQt},
}
Figure