TL;DR
Even though popular open LLM pretraining datasets (like C4, FineWeb, and RefinedWeb) are mined from the exact same CommonCrawl source with similar pipelines, neural classifiers can easily tell them apart—far outperforming human annotators. Crucially, these unique dataset "fingerprints" survive aggressive LLM paraphrasing and propagate into trained models, enabling classifiers to identify which dataset trained an LLM and accurately reverse-engineer pretraining data mixture ratios.
What Problem Does This Paper Address?
Most frontier open-weights language models are pretrained on filtered web corpora derived from CommonCrawl. While dataset creators describe these corpora as broad, diverse reflections of the open internet, their filtering recipes vary: some apply heuristic cleaning, others use perplexity filters, and modern sets deploy machine learning classifiers to isolate high-quality or educational text.
Borrowing inspiration from computer vision—where classic work revealed that "neutral" benchmark images carry distinctive dataset-level biases—the authors ask:
- Do standard web-text pretraining datasets carry identifiable statistical "fingerprints"?
- What features drive these biases?
- Do these dataset biases survive into model weights and manifest in LLM-generated text?
Methods
- Dataset Classification Framework: The authors frame bias detection as an $N$-way classification task. They adapt standard autoregressive transformers (primarily a 160M parameter model pretrained compute-optimally on 3.2B tokens) by swapping the language modeling head for an $N$-class linear projection, training with token-level cross-entropy loss across all sequence prefixes.
- Isolating Bias Sources: To determine whether classifiers rely merely on surface formatting or deeper semantic/lexical traits, they run several probing controls:
- LLM Rewriting: Rephrasing texts with GPT-4o-mini across three prompts that progressively strip formatting and alter sentence structure.
- Regex Normalization: Stripping newlines, lists, bullet points, tabs, and excess whitespace.
- Bag-of-Words (BoW): Classifying purely on word frequency distributions, discarding syntax and context.
- Thematic Profiling: Tagging sequences into 13 high-level categories (e.g., Science, News, Ads) using GPT-4o-mini.
- Bias Propagation & Mixture Auditing: They generate unconditioned text from LLMs trained exclusively on specific corpora (Falcon-7B, DCLM-7B, and FineWeb-Edu-1.8B) and test whether classifiers trained only on the raw pretraining data can identify the source model. They then apply this technique to models trained on multi-domain mixtures (SlimPajama) to estimate undisclosed data mixture proportions.
Datasets
The study evaluates seven open web corpora derived from CommonCrawl, categorized by filtering complexity:
- Category 1 (Heuristics + Deduplication): C4, FineWeb, and RefinedWeb.
- Category 2 (Category 1 + Wikipedia Perplexity Filtering): DolmaCC and RedPajama-V2.
- Category 3 (Category 2 + ML-Based Quality Filtering): DCLM-BASELINE and FineWeb-Edu.
- Multi-Domain Mixture: SlimPajama (and its 1.3B models trained on 4-domain and 7-domain splits spanning Arxiv, Books, C4, CC, Github, Stack Exchange, and Wikipedia).
Key Insights and Ideas
- Imperceptible to Humans, Obvious to Models: While human researchers struggle to distinguish C4 from FineWeb (scoring near chance), neural classifiers easily detect subtle pipeline artifacts.
- Biases Are More Than Formatting: While some artifacts are structural (e.g., the Resiliparse tool used by DCLM frequently inserts double newlines
\n\n), biases persist even after regex cleaning and LLM rewriting. BoW classifiers also beat random guessing, indicating persistent differences in vocabulary distributions. - Content Filtering Shifts Domain Proportions: ML-filtered datasets (Dolma, DCLM, FineWeb-Edu) substantially suppress advertising content. FineWeb-Edu skews heavily toward "Science" and "Education," making it uniquely identifiable.
- LLMs Faithfully Memorize and Leak Data Bias: Models generate text that mirrors the statistical quirks of their training corpora. A classifier trained strictly on original human web text identifies LLM-generated samples almost as accurately as the real text.
- Unsupervised Pretraining Mixture Reverse-Engineering: Because an LLM's unprompted generations statistically follow its training distribution, one can estimate the exact proportions of a training mixture by classifying a small sample (~2,000) of generated sequences.
Key Results
- Dataset Classification:
- In a 3-way classification of heuristic datasets (C4 vs. FineWeb vs. RefinedWeb), the classifier reaches 74.76% accuracy (chance: 33.3%).
- In 5-way classification across Categories 1 and 2, accuracy reaches 60.70% (chance: 20%).
- Categorization including ML-filtered data (DCLM, FineWeb-Edu) surpasses 84%–94% in 3-way setups.
- Human vs. Machine: On binary C4 vs. FineWeb classification, machine learning researchers averaged 63% accuracy (taking 1.5 hours for 100 samples), while the 160M model achieved 87.37% (and 88% in ablation settings).
- Robustness to Rewriting & Cleaning:
- Normalizing formatting with regex drops binary C4 vs. FineWeb accuracy from 87.37% to 72.42%—still far above random chance.
- Even under heavy LLM rewriting (Prompt 3), accuracy remains at 66.02%.
- Bias Propagation to LLMs:
- When distinguishing generated text from Falcon-7B, DCLM-7B, and FineWeb-Edu-1.8B using a 3-way classifier trained entirely on original web datasets, the classifier achieves 89.15% accuracy (nearly matching its 89.76% accuracy on real web text).
- Instruction tuning degrades this signal somewhat, but binary classification between Falcon-7B-Instruct and DCLM-7B-IT remains high at 89.09% (down from 97.39% for base models).
- Mixture Proportion Recovery: On 4-domain and 7-domain SlimPajama models, the classifier's predicted distribution of 2,048 random model completions closely mirrors the ground-truth domain mixtures.
Limitations and Future Work
- Shared Sub-Domains Confound Classifiers: The approach struggles when datasets contain identical underlying sources differing only by mixture weights. For example, because C4 is a filtered subset of CommonCrawl (CC), some generated CC text is misclassified as C4.
- Sequence Length Dependence: Classification accuracy scales with sequence length (climbing from ~71% on short sequences to >85% on 2,000-token documents).
- Supervised Fingerprinting Only: The methodology assumes access to samples of candidate pretraining datasets (or knowledge of potential training domains) to train the initial classifier. Extending this to purely unsupervised detection of completely unknown datasets remains open for future research.
Summary updated