TL;DR
Pretraining datasets for Large Language Models (LLMs), even when derived from the same web crawls and cleaned with similar pipelines, possess unique "fingerprints" that a simple classifier can detect with high accuracy. These signatures propagate through the training process, allowing researchers to identify a model's training sources and even detect potential data distillation between models like GPT-4o and DeepSeek-V3.
What Problem Does This Paper Address?
Most modern LLMs are trained on massive datasets scraped from the web, primarily sourced from CommonCrawl. While developers use various filtering, deduplication, and quality-control steps (like those used for C4, FineWeb, or RefinedWeb), it has been unclear whether these datasets remain distinct or become essentially interchangeable "generic web text."
The authors investigate whether these datasets have unique characteristics—"fingerprints"—that distinguish them. They also explore whether these fingerprints survive the training process, appearing in the text generated by the resulting models. This is crucial for understanding model generalization and for "forensic" auditing of closed-source models to determine their training data mixtures or potential reliance on synthetic data from other models.
Methods
The researchers employed several clever techniques to probe these fingerprints:
- Dataset Classification: They trained a 160M parameter autoregressive transformer to perform N-way classification, identifying which specific dataset a text sequence belongs to.
- Human Baseline: They compared the model's performance against human researchers to see if these differences are perceptible to people.
- Feature Analysis: To isolate what drives distinguishability, they used "rewrite" experiments (rephrasing text with GPT-4o-mini), "bag of words" models (ignoring structure), and regex-based formatting removal.
- Fingerprint Propagation: They tested if a classifier trained on original training data could correctly identify text generated by models trained on that data.
- Model Forensics: They analyzed the outputs of popular instruction-tuned models (like GPT-4o, Claude, and DeepSeek) to see if they share fingerprints, which might hint at data contamination or distillation.
Datasets
The study focused on seven major open-source pretraining datasets derived from CommonCrawl:
- Category 1 (Heuristic filtering): C4, FineWeb, RefinedWeb.
- Category 2 (Perplexity-based filtering): DolmaCC, RedPajama-V2.
- Category 3 (ML-based filtering): DCLM-Baseline, FineWeb-edu.
They also used benchmarks like WikiText-103 and Paloma for evaluation, and instruction-tuning prompt sets like OpenHermes-2.5, Alpaca, and UltraChat.
Key Insights and Ideas
- Fingerprints are pervasive and subtle: Even datasets with nearly identical curation steps are highly distinguishable. For example, C4 and FineWeb look very similar to humans, but a model can tell them apart with 88% accuracy.
- Formatting is a major signal: A significant part of a dataset's fingerprint comes from how it was parsed (e.g., how newlines or bullet points are handled). DCLM-Baseline is particularly distinct because its parser frequently inserts double newlines (
\n\n) between sentences. - The Generalization Penalty: Models trained on a single dataset generalize poorly to others. Training on a mixture of datasets significantly improves performance (lower perplexity) across the board, suggesting that "fingerprints" represent biases that can hinder a model's breadth if not balanced.
- Forensic Auditing: The most striking finding is that fingerprints propagate. You can estimate the mixture proportions of a model's training data just by classifying its random outputs.
- The DeepSeek/GPT-4o Connection: The authors found that responses from DeepSeek-V3 and GPT-4o to certain prompts are nearly indistinguishable to a classifier (48.7% error rate, essentially a coin flip). This provides strong evidence that DeepSeek-V3 likely used GPT-4o-generated data during its instruction finetuning.
Key Results
- High Classification Accuracy: Across various dataset combinations, the 160M classifier consistently achieved high accuracy. Even for the most similar datasets (Category 1), 3-way classification reached ~75% accuracy (vs. 33% chance).
- Mixture Benefits: Training on a mixture of datasets resulted in the best overall generalization, as shown in the perplexity table below. The "Mixture" model consistently outperformed single-dataset models on external benchmarks like WikiText-103 and Paloma.
- Thematic Differences: While similarly filtered datasets have comparable broad content distributions, Category 3 datasets (like FineWeb-edu) show a clear shift toward educational and scientific content, providing a thematic basis for their unique fingerprints.
Limitations and Future Work
The authors acknowledge a fundamental limitation: if two datasets share the exact same sources but only differ in their mixture proportions, a simple classifier might struggle to distinguish individual sequences accurately. For instance, since C4 is a subset of CommonCrawl (CC), some CC sequences are naturally classified as C4.
Future work could involve developing more sophisticated methods for mixture estimation in "nested" datasets and exploring how these fingerprints might be used to watermark or protect proprietary datasets from unauthorized use in model training.
Summary updated