Scaling Parameter-Constrained Language Models with Quality Data

Ernie Chang, Matteo Paltenghi, Yang Li, Pin-Jie Lin, Changsheng Zhao, Patrick Huber, Zechun Liu, Rastislav Rabatin, Yangyang Shi, Vikas Chandra

Abstract

Scaling laws in language modeling traditionally quantify training loss as a function of dataset size and model parameters, providing compute-optimal estimates but often neglecting the impact of data quality on model generalization. In this paper, we extend the conventional understanding of scaling law by offering a microscopic view of data quality within the original formulation—effective training tokens—which we posit to be a critical determinant of performance for parameter-constrained language models. Specifically, we formulate the proposed term of effective training tokens to be a combination of two readily-computed indicators of text: (i) text diversity and (ii) syntheticity as measured by a teacher model. We pretrained over 200 models of 25M to 1.5B parameters on a diverse set of sampled, synthetic data, and estimated the constants that relate text quality, model size, training tokens, and eight reasoning task accuracy scores. We demonstrated the estimated constants yield +0.83 Pearson correlation with true accuracies, and analyzed it in scenarios involving widely-used data techniques such as data sampling and synthesis which aim to improve data quality.

Introduction

The development of Large Language Models (LLMs) has been largely guided by scaling laws, such as the Chinchilla law, which suggest that model performance is a predictable function of the number of parameters ($N$) and the volume of training tokens ($D$). While these laws provide a robust framework for finding compute-optimal configurations, they typically treat all tokens as having equal value. This "quantity-first" perspective is increasingly problematic for parameter-constrained scenarios—such as on-device AI—where hardware limitations prevent simply increasing $N$.

In these contexts, the quality of the data becomes the primary lever for performance. This research introduces a revised scaling law that incorporates "effective training tokens" ($D_q$), a metric that adjusts raw token counts based on data diversity and syntheticity. By quantifying the impact of data quality, we provide a roadmap for building high-performance, compact models.

Methodology: Defining Data Quality

To move beyond raw token counts, we propose replacing $D$ in the standard scaling equation with $D_q = Q \cdot D$, where $Q$ represents a quality multiplier. We define $Q$ through two measurable indicators:

  1. Diversity: This is measured using the compression ratio via Gzip. A dataset that compresses heavily is considered repetitive and low-diversity, whereas less compressible data suggests higher information density.
  2. Syntheticity: This is measured by the "predictability" of the text according to a teacher model (e.g., Llama-2 7B). High syntheticity implies the data is clean, structured, and "textbook-like," which is often more conducive to learning reasoning patterns.

To validate this framework, we conducted an extensive experimental sweep, pretraining over 200 models ranging from 25 million to 1.5 billion parameters. We utilized two primary data refinement techniques: Coreset Selection (importance sampling to identify the most relevant data) and Data Synthesis (using Llama-3-70B to transform web content into high-quality reasoning problems).

Experimental Setup

The models were trained on subsets of RefinedWeb, a massive web-crawled dataset. Performance was evaluated across eight zero-shot reasoning benchmarks to calculate an average accuracy score. These benchmarks included ARC-easy, ARC-challenge, BoolQ, PIQA, SIQA, HellaSwag, OBQA, and WinoGrande.

Key Insights and Results

The Quality Multiplier Effect

Our findings suggest that data quality functions as a mathematical multiplier on dataset size. High-quality data effectively makes a dataset "feel" larger to the model. For instance, models trained on refined or synthetic data significantly outperformed those trained on randomly sampled raw web data across the entire parameter spectrum (25M to 1.5B).

The Diversity-Syntheticity Trade-off

We observed an inverse relationship between diversity and syntheticity. While synthetic data generated by LLMs is exceptionally clean and structured (high syntheticity), it tends to be less diverse than organic web text. The study identified a "sweet spot" for data mixing, where the benefits of high-quality synthetic structure are balanced against the breadth of natural language diversity.

Sensitivity of Small Models

The impact of data quality is most pronounced in smaller models (under 500M parameters). As models grow toward 1.5B parameters and beyond, the sheer quantity of tokens begins to regain its status as a dominant driver of performance. This suggests that for ultra-compact models, data curation is not just beneficial but essential.

Predictive Accuracy

The revised scaling formula achieved a +0.83 Pearson correlation with actual model accuracy. This high degree of correlation demonstrates that we can now predict the performance of parameter-constrained models based on the quality metrics of the training data before the training process even begins.

Conclusion

This research shifts the focus of scaling laws from pure volume to a more nuanced understanding of data quality. By introducing the concept of effective training tokens, we provide a quantitative method for evaluating data refinement techniques. Our results show that synthetic data is particularly effective at increasing the effective token count, though it must be balanced with diversity. Future work will explore task-specific quality measures to further refine these laws for specialized on-device applications.

Summary updated

Abstract

Scaling laws in language modeling traditionally quantify training loss as a function of dataset size and model parameters, providing compute-optimal estimates but often neglecting the impact of data quality on model generalization. In this paper, we extend the conventional understanding of scaling law by offering a microscopic view of data quality within the original formulation -- effective training tokens -- which we posit to be a critical determinant of performance for parameter-constrained language models. Specifically, we formulate the proposed term of effective training tokens to be a combination of two readily-computed indicators of text: (i) text diversity and (ii) syntheticity as measured by a teacher model. We pretrained over $200$ models of 25M to 1.5B parameters on a diverse set of sampled, synthetic data, and estimated the constants that relate text quality, model size, training tokens, and eight reasoning task accuracy scores. We demonstrated the estimated constants yield +0.83 Pearson correlation with true accuracies, and analyzed it in scenarios involving widely-used data techniques such as data sampling and synthesis which aim to improve data quality.
BibTeX
@inproceedings{chang2024scaling,
  author = {Ernie Chang and Matteo Paltenghi and Yang Li and Pin-Jie Lin and Changsheng Zhao and Patrick Huber and Zechun Liu and Rastislav Rabatin and Yangyang Shi and Vikas Chandra},
  title = {{Scaling Parameter-Constrained Language Models with Quality Data}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=oIsnNXSOKI},
}
Figure