TL;DR
As frontier models exhaust the supply of human-written text, synthetic data has become essential for pre-training—yet measuring whether generated data is genuinely diverse remains an open problem. This paper introduces LLM Cluster-agent, a clustering-based metric that accurately quantifies text diversity where standard statistical metrics fail, and demonstrates through controlled pre-training runs (350M and 1.4B parameters) that higher synthetic data diversity directly fuels both pre-training and downstream fine-tuning gains.
What Problem Does This Paper Address?
High-quality web text is rapidly becoming scarce, pushing LLM developers toward synthetic data (such as Microsoft's Phi series and Hugging Face's Cosmopedia). However, simply scaling synthetic token volume can cause models to collapse if the data lacks diversity.
Traditional text diversity metrics—such as vocabulary size, $n$-gram uniqueness, compression ratios, and embedding-based $K$-means clustering—fail to capture deeper semantic variety and often show conflicting or misleading trends when applied to machine-generated text. Researchers lacked both a reliable metric to quantify synthetic text diversity and a rigorous, controlled study examining how different dimensions of synthetic data (topic distributions, prompt conditioning, generator models, and real-to-synthetic ratios) impact downstream pre-training and supervised fine-tuning (SFT).
Methods
To reliably evaluate diversity, the authors develop LLM Cluster-agent, a multi-step pipeline that leverages an LLM's semantic reasoning:
- Metadata & Metric Generation: The agent inspects small sample batches ($J=5$, repeated $M=100$ times) and induces 3–5 dataset-specific clustering criteria and 1–5 scoring rules (e.g., conceptual density, disciplinary focus).
- Iterative Clustering: Random batches of $K=10$ samples are clustered across $N=5{,}000$ iterations along with explanatory rationales.
- Self-Verification: A verification prompt checks the validity of generated clusters, filtering out spurious groupings.
- Scoring: The final diversity score $D$ is the average ratio of valid clusters $C_i$ to sample count $S_i$: $$D = \frac{1}{N} \sum_{i=1}^N \frac{C_i}{S_i}$$
Using this metric, the authors conduct controlled pre-training runs with Llama-architecture models (350M and 1.4B parameters) trained on up to 150B tokens, testing the effects of topic breadth, prompt engineering, generator choice, and real-to-synthetic token mixing ratios.
Datasets
- Real Pre-training Corpus (34B tokens): A 4:1:1 blend of filtered web data (Wikipedia and C4 subsets of Dolma), code (The Stack, StackOverflow, CodeContests), and math (OpenWebMath).
- Synthetic Pre-training Datasets: Generated from a curated hierarchy of 620,000 Wikipedia topic seeds using varying prompts (standard Topic, Topic Styles, Topic Styles Persona, and Multi-Topic). Baselines include Cosmopedia v0.1 and v0.2.
- SFT Corpus: A combination of GPT-4-filtered Alpaca and FLANv2 (trained for 3 epochs).
- Evaluation Benchmarks: 8 zero-shot commonsense and language understanding tasks evaluated via LM-Eval Harness: ARC-Easy, ARC-Challenge, BoolQ, SIQA, WinoGrande, PIQA, HellaSwag, and COPA.
Key Insights and Ideas
- Standard diversity metrics fail on synthetic text: Metrics like $n$-gram diversity and embedding $K$-means produce flat or counterintuitive trends. For instance, $K$-means scored 300K topics as less diverse than 100K topics, whereas LLM Cluster-agent accurately reflected the true underlying distributions.
- More generations per topic hits diminishing returns: Increasing generations per topic ($G$) from 10 to 20 improves performance, but pushing to $G=30$ induces redundant syntax and content, decreasing the LLM cluster score and degrading benchmark performance.
- Personas and styles unlock real diversity: Conditioning data generation on varied styles (textbook, narrative, blog post, wikihow) and diverse target audience personas significantly outperforms standard textbook prompts. Interestingly, packing multiple topics into a single generation (Multi-Topic) reduced diversity by introducing structural redundancy.
- Pre-training diversity matters most downstream: Even when smaller models (350M) appear to saturate during pre-training, higher pre-training diversity continues to yield substantial improvements after supervised fine-tuning.
- Balance is crucial: Skewing the training blend heavily toward synthetic tokens (e.g., beyond 34B synthetic against 34B real) sharply degrades performance.
Key Results
- Strong Performance Correlation: The LLM Cluster score exhibits a strong positive Pearson correlation with downstream accuracy ($r = 0.7481, p=0.0051$ for 350M SFT; $r = 0.8457, p=0.0005$ for 1.4B pre-training), easily outperforming existing heuristics.
- Beating Established Baselines: Models trained on the Topic Styles Persona synthetic data achieved 57.82% pre-train and 61.32% SFT average accuracy at 1.4B scale, significantly beating the real-only baseline (54.76% / 57.16%) as well as Cosmopedia v0.1 (56.25% / 59.46%) and v0.2 (55.84% / 59.46%) using fewer total synthetic tokens.
- Generator Capability Matters: Stronger base generators produced higher diversity scores and superior downstream models (GPT-4o > GPT-3.5; Llama-3.1-8B > Mistral-7B). Blending synthetic data across multiple generators yielded the highest 350M scores (51.72% pre-train, 53.02% SFT).
Limitations and Future Work
- Model Scale: Due to compute constraints, the study evaluated models up to 1.4B parameters and pre-training budgets up to 150B tokens; extending these findings to 7B+ parameters and multi-trillion-token regimes remains future work.
- Domain and Language Scope: The synthetic data generation relied on English Wikipedia topics and general web text; applying this framework to domain-specific math or code generation requires further exploration.
- Metric Computation Cost: Because LLM Cluster-agent relies on iterative LLM API queries, measuring massive corpora requires subsampling/bootstrapping ($K=10$, $N=5{,}000$).
Summary updated