TL;DR
Standard scaling laws assume that all training tokens are created equal, focusing solely on model size and raw data quantity. This paper revises the Chinchilla scaling law by introducing effective training tokens—a quality-adjusted token metric based on text diversity and model-derived "syntheticity." By training over 200 small-to-medium language models (25M to 1.5B parameters), the authors demonstrate that data quality dramatically boosts downstream reasoning performance for parameter-constrained models, predicting zero-shot accuracy with a +0.83 Pearson correlation.
What Problem Does This Paper Address?
Classical scaling laws (e.g., Kaplan et al., Hoffmann et al. / Chinchilla) model training loss purely as a function of parameter count ($N$) and token quantity ($D$). However, this quantity-centric framework overlooks data quality, which is especially critical when training parameter-constrained models (sub-billion parameters) targeted for on-device deployment.
Modern pretraining increasingly relies on data curation techniques like deduplication, importance sampling, and synthetic data generation. The authors set out to quantify how these data refinement strategies impact scaling dynamics and downstream common-sense reasoning performance.
Methods
The authors reformulate the scaling law by replacing raw token count $D$ with a quality-adjusted token count, $D_q = D \cdot Q$, where $Q$ is a scaling factor:
$$Q = \exp\left(c_1 \cdot \text{diversity} + c_2 \cdot \text{syntheticity}\right)$$
They measure quality using two scalable, data-agnostic metrics:
- Diversity ($Dr(D)$): Defined as the inverse compression ratio ($CR(D)^{-1}$) computed via
gzip. Less redundant, more diverse text compresses poorly, yielding a higher diversity score. - Syntheticity ($S(D)$): Defined as the inverse perplexity under a reference teacher model (Llama-2 7B Chat). Sequences that are highly predictable under the teacher model have low perplexity and therefore high syntheticity.
Instead of modeling pretraining loss, the authors adapt the scaling formula to predict zero-shot downstream accuracy $\hat{G}(N, D)$ across eight reasoning tasks:
$$\hat{G}(N, D) = \text{clip}\left(E + \frac{A}{N^\alpha} + \frac{B}{D_q^\beta}\right)$$
Constants ($A, B, E, \alpha, \beta, c_1, c_2$) were fit using non-linear least squares across 210 experimental runs.
Datasets
- Pretraining Base Corpus: RefinedWeb, partitioned into three variations:
- Random: Randomly sampled subsets from 10% to 100% (up to 8B tokens).
- Selected: Coreset selection via $n$-gram importance sampling matched against evaluation targets (up to 7B tokens).
- Synthetic: Paraphrased pretraining text transformed into question-answer reasoning pairs or textbook-style prose using Llama-3-70B-Instruct (up to 2B tokens).
- Evaluation Benchmarks: Eight standard zero-shot common-sense reasoning benchmarks: ARC-Easy, ARC-Challenge, BoolQ, PIQA, SIQA, HellaSwag, OpenBookQA (OBQA), and WinoGrande.
Key Insights and Ideas
- Syntheticity vs. Diversity Trade-off: Synthetic data generated by teacher LLMs typically exhibits lower diversity (it compresses easily) but significantly higher syntheticity (highly typical/predictable phrasing).
- The Paradox of Diversity in Reasoning: Fitting the scaling formula yielded a negative diversity coefficient ($c_1 = -12.78$) and a positive syntheticity coefficient ($c_2 = 0.64$). In this setup, clean, predictable, textbook-like structure (lower lexical entropy, higher predictability) substantially elevates the quality multiplier $Q$ for downstream reasoning compared to messy, highly diverse web text.
- Data Quality is Token-Quantity Bound: For small models (25M to 500M), data quality ($Q$) exerts an outsized influence on performance. However, as model size approaches and exceeds 1.5B parameters, the models become "token-hungry," and raw token volume reasserts itself as the primary scaling constraint.
Key Results
- High Predictive Correlation: The revised scaling formulation achieved a +0.83 Pearson correlation between predicted accuracy $\hat{G}(N, D)$ and empirical evaluation scores across varying model sizes and data distributions.
- Fitted Parameters: Estimated values were $E = 1.1400$, $A = -0.8546$, $B = -18.3078$, $\alpha = 0.0450$, $\beta = 0.3683$, $c_1 = -12.78$, and $c_2 = 0.64$.
- Refinement Effectiveness: Synthetic data generation yielded a substantially steeper increase in effective tokens ($D_q$) per raw token trained than either selective sampling or raw web data.
Limitations and Future Work
- High Fitting Cost: Accurately estimating the constants required pretraining over 200 distinct model checkpoints; replicating this framework across different architectures or task domains entails massive computational overhead.
- Syntheticity Bottlenecks: Evaluating text syntheticity requires running inference with a teacher LLM across a substantial fraction of documents (25% in this work), which is computationally demanding at web scale.
- Scope of Model Scale: Empirical validations were restricted to models up to 1.5B parameters. Whether the specific interplay between compression ratio and teacher perplexity holds identically at the 7B–70B+ scale remains an open question.
Summary updated