Scaling Laws for Downstream Task Performance of Large Language Models

Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, Sanmi Koyejo

Introduction

Scaling laws have traditionally focused on "upstream" metrics, specifically predicting how pretraining cross-entropy loss decreases with increased data and parameters. However, the ultimate goal of large language models (LLMs) is performance on downstream tasks—such as translation, coding, or reasoning—after finetuning. This paper addresses the gap by proposing a new log-law to predict task-specific utility and identifying "alignment" as the primary driver of scaling behavior.

Methodology

The researchers utilized the T5 encoder-decoder architecture (770M and 3B parameters) to conduct systematic experiments:

  1. Controlled Pretraining: Models were pretrained on various mixtures of the MC4 dataset, varying language composition to control alignment with downstream tasks.
  2. Finetuning: Models were finetuned on WMT machine translation datasets (en-de, en-fr, en-ro).
  3. The Log-Law: A new functional form was proposed for task metrics: $f(D_p) = (\log(A \cdot D_p^\alpha))^\beta$, where $D_p$ is the pretraining data size.
  4. Alignment Score: A formal "Translation Alignment Score" was introduced to quantify the overlap between pretraining and downstream data distributions.

The Cross-Entropy Mirage and the Role of Alignment

A critical finding of this work is that cross-entropy loss can be a deceptive proxy for actual model utility. While cross-entropy loss typically improves monotonically with more data, task-specific metrics like BLEU, ROUGE, or COMET can fluctuate or even degrade if the pretraining data is poorly aligned with the downstream task.

Scaling of task metrics (BLEU, ROUGE, COMET) vs. cross-entropy loss for more and less aligned data.

As shown in the figure above, "well-aligned" data (orange) follows a predictable upward trajectory in task scores. In contrast, "less aligned" data (red) shows non-monotonic behavior in BLEU and ROUGE scores, despite the cross-entropy validation loss continuing to decrease. This highlights that alignment is the deciding factor in whether additional pretraining compute translates to better downstream performance.

Key Results

  • Predictive Accuracy: In well-aligned settings, the proposed log-law fits empirical results with high precision (prediction error $\le 0.061$ for BLEU).
  • Power Law for Downstream CE: While task scores follow a log-law, downstream cross-entropy still follows a power-law, mirroring upstream behavior.
  • Generalization: The log-law successfully described scaling for non-translation tasks in SuperGLUE (e.g., BoolQ, COPA, RTE), suggesting it may be a universal property of task-specific accuracy.
  • The Alignment Penalty: For an English-to-French task, pretraining on 100% English data (Alignment = 0.7) led to lower BLEU scores and more "broken" scaling compared to a 50/50 English-French mix (Alignment = 1.0).
  • Diminishing Returns: If the finetuning dataset is sufficiently large, the impact of pretraining data size diminishes significantly.

Conclusion and Future Work

This research provides a practical heuristic for developers: pretrain on a small data subset, fit the log-law coefficients, and use the results to determine the value of continued pretraining on a specific data mixture.

However, limitations remain. The current Alignment Score is specific to translation, and defining a universal alignment score for abstract tasks like reasoning is an open challenge. Furthermore, while the laws hold up to 3B parameters, further testing is required to see if they persist for frontier models at the 100B+ scale.

Summary updated

BibTeX
@inproceedings{isik2024scaling,
  author = {Berivan Isik and Natalia Ponomareva and Hussein Hazimeh and Dimitris Paparas and Sergei Vassilvitskii and Sanmi Koyejo},
  title = {{Scaling Laws for Downstream Task Performance of Large Language Models}},
  booktitle = {DPFM @ ICLR},
  year = {2024},
  url = {https://openreview.net/forum?id=PXwdsgZjnX},
}
Figure