On the Diversity of Synthetic Data and its Impact on Training Large Language Models

Hao Chen, Abdul Waheed, Xiang Li, Yidong Wang, Jindong Wang, Bhiksha Raj, Marah I Abdin

Abstract

The rise of Large Language Models (LLMs) has accentuated the need for diverse, high-quality pre-training data. Synthetic data emerges as a viable solution to the challenges of data scarcity and inaccessibility. While previous literature has focused predominantly on the quality and quantity of real data, our work enables the measurement of diversity in synthetic data and explores its impact on LLM performance. We study the downstream effects of synthetic data diversity during both the pre-training and fine-tuning stages by introducing a new diversity metric, \textit{LLM cluster-agent}, designed to evaluate the diversity of synthetic datasets. Through a series of controlled experiments with models of 350M and 1.4B parameters, we demonstrate that the proposed cluster-based LLM scoring of diversity correlates positively with both pre-training and supervised fine-tuning performance. Our findings also reveal that synthetic data diversity in pre-training affects supervised fine-tuning more significantly than pre-training itself, even for smaller models. We hope this study advances our understanding of the optimal use of synthetic data in LLM training and opens new avenues for efficient data generation processes.

Introduction

As the AI community approaches the "data wall"—the point where high-quality human-generated text on the internet is exhausted—researchers are increasingly turning to synthetic data to train the next generation of Large Language Models (LLMs). However, synthetic data often suffers from a lack of variety, leading to repetitive patterns that can hinder a model's reasoning and generalization. Traditional heuristic metrics like n-gram overlap or perplexity are shallow and fail to capture the underlying semantic breadth of a dataset. This paper introduces a reliable way to measure semantic diversity and investigates how that diversity impacts model performance during both pre-training and fine-tuning.

Methodology: The LLM Cluster-agent Metric

To quantify semantic diversity, the authors propose LLM Cluster-agent, a three-stage pipeline:

  1. Metadata & Metric Generation: An LLM (e.g., GPT-4o) analyzes a small sample of the data to define relevant "clustering criteria" such as disciplinary focus, conceptual density, or narrative style.
  2. Iterative Clustering: The LLM then groups batches of data points based on these criteria, providing reasoning for each cluster.
  3. Self-Verification: A final verification step ensures the clusters are semantically consistent, discarding unreasonable groupings.

The final LLM Cluster Score is the average number of valid clusters found per sample, offering a more accurate semantic measurement than statistical heuristics.

Experimental Framework

The study involved pre-training models with 350M and 1.4B parameters on a mix of 34B real tokens and billions of synthetic tokens. Synthetic data was generated using 620,000 Wikipedia topics as seeds, with variations in topic count, generations per topic, and prompt styles. Prompting strategies ranged from simple textbook styles to complex "Personas" (e.g., writing as a specific expert for a specific audience). Models were evaluated on standard benchmarks including WinoGrande, ARC, BoolQ, and HellaSwag.

Results: The Impact of Diversity

The experimental results demonstrate that synthetic data diversity is a critical predictor of model performance.

Correlation between LLM Cluster Score and Average Accuracy across model sizes (350M, 1.4B) and training stages (Pre-train, SFT).

Key insights from the data include:

  • Strong Correlation: The proposed LLM Cluster Score showed a high positive correlation with benchmark accuracy, far outperforming traditional metrics like n-gram diversity or K-means clustering.
  • The Persona Power-up: Instructing the generator to adopt different "personas" was one of the most effective ways to increase synthetic diversity, outperforming more complex multi-topic generation strategies.
  • Diversity as an SFT Catalyst: Diversity in pre-training data has a significantly larger impact on Supervised Fine-Tuning (SFT) than on the pre-training stage itself. A diverse pre-training foundation makes the model more "teachable" during subsequent instruction tuning.
  • Scaling Effects: While smaller models (350M) saw their pre-training performance saturate quickly, the 1.4B models showed a steeper performance climb as data diversity increased, suggesting that larger models are better equipped to absorb complex, diverse information.

Conclusion and Future Work

This study establishes that the semantic diversity of synthetic data, as measured by the LLM Cluster-agent metric, is a vital factor in LLM performance. While the study was limited to models up to 1.4B parameters, the findings suggest that optimizing for diversity is key to overcoming the data wall. Future research will explore these scaling laws at larger magnitudes and investigate more efficient ways to generate diverse synthetic corpora.

Summary updated

Abstract

The rise of Large Language Models (LLMs) has accentuated the need for diverse, high-quality pre-training data. Synthetic data emerges as a viable solution to the challenges of data scarcity and inaccessibility. While previous literature has focused predominantly on the quality and quantity of real data, our work enables the measurement of diversity in synthetic data and explores its impact on LLM performance. We study the downstream effects of synthetic data diversity during both the pre-training and fine-tuning stages by introducing a new diversity metric, \textit{LLM cluster-agent}, designed to evaluate the diversity of synthetic datasets. Through a series of controlled experiments with models of 350M and 1.4B parameters, we demonstrate that the proposed cluster-based LLM scoring of diversity correlates positively with both pre-training and supervised fine-tuning performance. Our findings also reveal that synthetic data diversity in pre-training affects supervised fine-tuning more significantly than pre-training itself, even for smaller models. We hope this study advances our understanding of the optimal use of synthetic data in LLM training and opens new avenues for efficient data generation processes.
BibTeX
@misc{chen2024diversity,
  author = {Hao Chen and Abdul Waheed and Xiang Li and Yidong Wang and Jindong Wang and Bhiksha Raj and Marah I Abdin},
  title = {{On the Diversity of Synthetic Data and its Impact on Training Large Language Models}},
  year = {2024},
  eprint = {2410.15226},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.15226},
}

Loading related content...

Figure