Scaling Parameter-Constrained Language Models with Quality Data

Ernie Chang, Matteo Paltenghi, Yang Li, Pin-Jie Lin, Changsheng Zhao, Patrick Huber, Zechun Liu, Rastislav Rabatin, Yangyang Shi, Vikas Chandra

TL;DR

Standard scaling laws assume that all training tokens are created equal, focusing solely on model size and raw data quantity. This paper revises the Chinchilla scaling law by introducing effective training tokens—a quality-adjusted token metric based on text diversity and model-derived "syntheticity." By training over 200 small-to-medium language models (25M to 1.5B parameters), the authors demonstrate that data quality dramatically boosts downstream reasoning performance for parameter-constrained models, predicting zero-shot accuracy with a +0.83 Pearson correlation.

What Problem Does This Paper Address?

Classical scaling laws (e.g., Kaplan et al., Hoffmann et al. / Chinchilla) model training loss purely as a function of parameter count ($N$) and token quantity ($D$). However, this quantity-centric framework overlooks data quality, which is especially critical when training parameter-constrained models (sub-billion parameters) targeted for on-device deployment.

Modern pretraining increasingly relies on data curation techniques like deduplication, importance sampling, and synthetic data generation. The authors set out to quantify how these data refinement strategies impact scaling dynamics and downstream common-sense reasoning performance.

Methods

The authors reformulate the scaling law by replacing raw token count $D$ with a quality-adjusted token count, $D_q = D \cdot Q$, where $Q$ is a scaling factor:

$$Q = \exp\left(c_1 \cdot \text{diversity} + c_2 \cdot \text{syntheticity}\right)$$

They measure quality using two scalable, data-agnostic metrics:

  1. Diversity ($Dr(D)$): Defined as the inverse compression ratio ($CR(D)^{-1}$) computed via gzip. Less redundant, more diverse text compresses poorly, yielding a higher diversity score.
  2. Syntheticity ($S(D)$): Defined as the inverse perplexity under a reference teacher model (Llama-2 7B Chat). Sequences that are highly predictable under the teacher model have low perplexity and therefore high syntheticity.

Instead of modeling pretraining loss, the authors adapt the scaling formula to predict zero-shot downstream accuracy $\hat{G}(N, D)$ across eight reasoning tasks:

$$\hat{G}(N, D) = \text{clip}\left(E + \frac{A}{N^\alpha} + \frac{B}{D_q^\beta}\right)$$

Constants ($A, B, E, \alpha, \beta, c_1, c_2$) were fit using non-linear least squares across 210 experimental runs.

Datasets

  • Pretraining Base Corpus: RefinedWeb, partitioned into three variations:
  • Random: Randomly sampled subsets from 10% to 100% (up to 8B tokens).
  • Selected: Coreset selection via $n$-gram importance sampling matched against evaluation targets (up to 7B tokens).
  • Synthetic: Paraphrased pretraining text transformed into question-answer reasoning pairs or textbook-style prose using Llama-3-70B-Instruct (up to 2B tokens).
  • Evaluation Benchmarks: Eight standard zero-shot common-sense reasoning benchmarks: ARC-Easy, ARC-Challenge, BoolQ, PIQA, SIQA, HellaSwag, OpenBookQA (OBQA), and WinoGrande.

Key Insights and Ideas

  • Syntheticity vs. Diversity Trade-off: Synthetic data generated by teacher LLMs typically exhibits lower diversity (it compresses easily) but significantly higher syntheticity (highly typical/predictable phrasing).
  • The Paradox of Diversity in Reasoning: Fitting the scaling formula yielded a negative diversity coefficient ($c_1 = -12.78$) and a positive syntheticity coefficient ($c_2 = 0.64$). In this setup, clean, predictable, textbook-like structure (lower lexical entropy, higher predictability) substantially elevates the quality multiplier $Q$ for downstream reasoning compared to messy, highly diverse web text.
  • Data Quality is Token-Quantity Bound: For small models (25M to 500M), data quality ($Q$) exerts an outsized influence on performance. However, as model size approaches and exceeds 1.5B parameters, the models become "token-hungry," and raw token volume reasserts itself as the primary scaling constraint.

Key Results

  • High Predictive Correlation: The revised scaling formulation achieved a +0.83 Pearson correlation between predicted accuracy $\hat{G}(N, D)$ and empirical evaluation scores across varying model sizes and data distributions.
  • Fitted Parameters: Estimated values were $E = 1.1400$, $A = -0.8546$, $B = -18.3078$, $\alpha = 0.0450$, $\beta = 0.3683$, $c_1 = -12.78$, and $c_2 = 0.64$.
  • Refinement Effectiveness: Synthetic data generation yielded a substantially steeper increase in effective tokens ($D_q$) per raw token trained than either selective sampling or raw web data.

Limitations and Future Work

  • High Fitting Cost: Accurately estimating the constants required pretraining over 200 distinct model checkpoints; replicating this framework across different architectures or task domains entails massive computational overhead.
  • Syntheticity Bottlenecks: Evaluating text syntheticity requires running inference with a teacher LLM across a substantial fraction of documents (25% in this work), which is computationally demanding at web scale.
  • Scope of Model Scale: Empirical validations were restricted to models up to 1.5B parameters. Whether the specific interplay between compression ratio and teacher perplexity holds identically at the 7B–70B+ scale remains an open question.

Summary updated

Abstract

Scaling laws in language modeling traditionally quantify training loss as a function of dataset size and model parameters, providing compute-optimal estimates but often neglecting the impact of data quality on model generalization.In this paper, we extend the conventional understanding of scaling law by offering a microscopic view of data quality within the original formulation – effective training tokens – which we posit to be a critical determinant of performance for parameter-constrained language models.Specifically, we formulate the proposed term of effective training tokens to be a combination of two readily-computed indicators of text:(i) text diversity and (ii) syntheticity as measured by a teacher model.We pretrained over 200 models of 25M to 1.5B parameters on a diverse set of sampled, synthetic data, and estimated the constants that relate text quality, model size, training tokens, and eight reasoning task accuracy scores.We demonstrated the estimated constants yield +0.83 Pearson correlation with true accuracies, and analyze it in scenarios involving widely-used data techniques such as data sampling and synthesis which aim to improve data quality.
BibTeX
@inproceedings{chang2024scaling,
  author = {Ernie Chang and Matteo Paltenghi and Yang Li and Pin-Jie Lin and Changsheng Zhao and Patrick Huber and Zechun Liu and Rastislav Rabatin and Yangyang Shi and Vikas Chandra},
  title = {{Scaling Parameter-Constrained Language Models with Quality Data}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=oIsnNXSOKI},
}

Loading related content...

Figure