Abstract
We observe that the development cross-entropy loss of supervised neural machine translation models scales like a power law with the amount of training data and the number of non-embedding parameters in the model. We discuss some practical implications of these results, such as predicting BLEU achieved by large scale models and predicting the ROI of labeling data in low-resource language pairs.
Introduction
Training state-of-the-art Neural Machine Translation (NMT) models has become a multi-million dollar engineering challenge. Organizations often face a difficult question: Is it better to spend money on more compute (larger models) or more data (human labeling)?
While previous research established "scaling laws" for unsupervised language models, it wasn't clear if these same mathematical trends held for supervised translation tasks. Furthermore, researchers needed a way to bridge the gap between "cross-entropy loss" (what the model optimizes) and "BLEU scores" (how humans actually measure translation quality). This work provides the mathematical framework to turn NMT scaling from guesswork into a predictable engineering discipline.
Scaling Methodology
The authors trained a wide array of Transformer models, varying both the number of non-embedding parameters ($N$) and the amount of training data ($D$). They adapted a specific power-law equation to model how cross-entropy loss ($L$) behaves as $N$ and $D$ increase.
Key components of the methodology include:
- The Scaling Ansatz: Adapting power-law equations to model loss behavior.
- The BLEU Bridge: Discovering a predictable exponential relationship between BLEU scores and cross-entropy loss: $BLEU \approx Ce^{-kL}$.
- Extrapolation: Using small models on tiny subsets of data (as little as 3%) to predict the performance of models orders of magnitude larger.
- Vocabulary Tuning: Investigating how standard Byte Pair Encoding (BPE) vocabulary sizes (e.g., 30k) can cause models to plateau on small datasets, and how smaller vocabularies (e.g., 2k) can restore scaling predictability.
Key Insights
- Predictability from Scraps: Using just 3% to 6% of a dataset is sufficient to estimate the scaling exponents that govern the full-scale system.
- The "Unigram Plateau": On extremely small datasets (<5MB), models rely on simple word-frequency statistics, causing power laws to fail. Shrinking the BPE vocabulary fixes this.
- Universal Exponents: Scaling exponents ($\alpha_N \approx 0.11–0.13$ and $\alpha_D \approx 0.35–0.43$) were remarkably similar across different language pairs, suggesting fundamental properties of the Transformer architecture and language data.
- ROI of Data Collection: By combining scaling laws with human translation costs, the authors created "USD-to-BLEU" projections, allowing for informed decisions on data labeling investments.
Results
For all tested languages, cross-entropy loss followed the power law with high precision. The exponential fit between loss and BLEU was highly accurate for BLEU scores above 15.
The results confirm that in many low-resource scenarios, data—not model size—is the primary bottleneck. Doubling the data leads to a much more significant gain than doubling the parameters.
Limitations and Future Work
While BLEU is predictable, high scores on specific test sets don't necessarily mean a model has "solved" translation, particularly across domains. Additionally, while the paper provides empirical proof of scaling, the underlying causal mechanisms remain a subject for future investigation, such as potential links to the "intrinsic dimension" of languages. Projections for extremely low-resource languages (under 40k lines) also remain less reliable than for mid-to-high resource pairs.
Summary updated