Abstract
Scaling laws in language modeling traditionally quantify training loss as a function of dataset size and model parameters, providing compute-optimal estimates but often neglecting the impact of data quality on model generalization. In this paper, we extend the conventional understanding of scaling law by offering a microscopic view of data quality within the original formulation—effective training tokens—which we posit to be a critical determinant of performance for parameter-constrained language models. Specifically, we formulate the proposed term of effective training tokens to be a combination of two readily-computed indicators of text: (i) text diversity and (ii) syntheticity as measured by a teacher model. We pretrained over 200 models of 25M to 1.5B parameters on a diverse set of sampled, synthetic data, and estimated the constants that relate text quality, model size, training tokens, and eight reasoning task accuracy scores. We demonstrated the estimated constants yield +0.83 Pearson correlation with true accuracies, and analyzed it in scenarios involving widely-used data techniques such as data sampling and synthesis which aim to improve data quality.
Introduction
The development of Large Language Models (LLMs) has been largely guided by scaling laws, such as the Chinchilla law, which suggest that model performance is a predictable function of the number of parameters ($N$) and the volume of training tokens ($D$). While these laws provide a robust framework for finding compute-optimal configurations, they typically treat all tokens as having equal value. This "quantity-first" perspective is increasingly problematic for parameter-constrained scenarios—such as on-device AI—where hardware limitations prevent simply increasing $N$.
In these contexts, the quality of the data becomes the primary lever for performance. This research introduces a revised scaling law that incorporates "effective training tokens" ($D_q$), a metric that adjusts raw token counts based on data diversity and syntheticity. By quantifying the impact of data quality, we provide a roadmap for building high-performance, compact models.
Methodology: Defining Data Quality
To move beyond raw token counts, we propose replacing $D$ in the standard scaling equation with $D_q = Q \cdot D$, where $Q$ represents a quality multiplier. We define $Q$ through two measurable indicators:
- Diversity: This is measured using the compression ratio via Gzip. A dataset that compresses heavily is considered repetitive and low-diversity, whereas less compressible data suggests higher information density.
- Syntheticity: This is measured by the "predictability" of the text according to a teacher model (e.g., Llama-2 7B). High syntheticity implies the data is clean, structured, and "textbook-like," which is often more conducive to learning reasoning patterns.
To validate this framework, we conducted an extensive experimental sweep, pretraining over 200 models ranging from 25 million to 1.5 billion parameters. We utilized two primary data refinement techniques: Coreset Selection (importance sampling to identify the most relevant data) and Data Synthesis (using Llama-3-70B to transform web content into high-quality reasoning problems).
Experimental Setup
The models were trained on subsets of RefinedWeb, a massive web-crawled dataset. Performance was evaluated across eight zero-shot reasoning benchmarks to calculate an average accuracy score. These benchmarks included ARC-easy, ARC-challenge, BoolQ, PIQA, SIQA, HellaSwag, OBQA, and WinoGrande.
Key Insights and Results
The Quality Multiplier Effect
Our findings suggest that data quality functions as a mathematical multiplier on dataset size. High-quality data effectively makes a dataset "feel" larger to the model. For instance, models trained on refined or synthetic data significantly outperformed those trained on randomly sampled raw web data across the entire parameter spectrum (25M to 1.5B).
The Diversity-Syntheticity Trade-off
We observed an inverse relationship between diversity and syntheticity. While synthetic data generated by LLMs is exceptionally clean and structured (high syntheticity), it tends to be less diverse than organic web text. The study identified a "sweet spot" for data mixing, where the benefits of high-quality synthetic structure are balanced against the breadth of natural language diversity.
Sensitivity of Small Models
The impact of data quality is most pronounced in smaller models (under 500M parameters). As models grow toward 1.5B parameters and beyond, the sheer quantity of tokens begins to regain its status as a dominant driver of performance. This suggests that for ultra-compact models, data curation is not just beneficial but essential.
Predictive Accuracy
The revised scaling formula achieved a +0.83 Pearson correlation with actual model accuracy. This high degree of correlation demonstrates that we can now predict the performance of parameter-constrained models based on the quality metrics of the training data before the training process even begins.
Conclusion
This research shifts the focus of scaling laws from pure volume to a more nuanced understanding of data quality. By introducing the concept of effective training tokens, we provide a quantitative method for evaluating data refinement techniques. Our results show that synthetic data is particularly effective at increasing the effective token count, though it must be balanced with diversity. Future work will explore task-specific quality measures to further refine these laws for specialized on-device applications.
Summary updated