Unified Scaling Laws for Routed Language Models

Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake A. Hechtman, Trevor Cai, Sebastian Borgeaud, George van den Driessche, Eliza Rutherford, Tom Hennigan, Matthew Johnson, Katie Millican, Albin Cassirer, Chris Jones, Elena Buchatskaya, David Budden, Laurent Sifre, Simon Osindero, Oriol Vinyals, Jack W. Rae, Erich Elsen, Koray Kavukcuoglu, Karen Simonyan

TL;DR

This paper introduces unified scaling laws for "routed" language models, which use only a subset of their parameters per input. These laws show that performance scales with both model size and the number of "experts" (parameter subsets), offering a more nuanced understanding than traditional scaling laws. This work provides a framework for comparing different routing techniques and designing more efficient large language models.

What Problem Does This Paper Address?

For years, the scaling of neural networks, particularly language models, has been observed to follow power-law relationships: as model size (parameter count) increases, performance (typically measured by loss) decreases predictably. This has been a cornerstone for building larger and more capable models. However, a new class of architectures called "Routing Networks" (or Mixture-of-Experts models) complicates this picture. These models use a "router" to select a subset of parameters (experts) for each input, meaning parameter count and computational cost are no longer directly coupled. This disconnect makes traditional scaling laws insufficient. The authors aim to develop new scaling laws that accurately describe these routed models, generalize existing ones, and provide a unified understanding of their performance across different architectures and training methods.

Methods

The core of this paper is the derivation and validation of new scaling laws for routed language models. They propose a bilinear scaling law that considers both the dense model size ($N$) and the number of experts ($E$), and crucially, incorporates a saturating transformation for $E$ to account for diminishing returns. This law is then generalized to use inference compute ($F$) and parameter utilization ratio ($B$) as variables, allowing for a unified scaling law across different routing architectures (varying $K$, the number of experts used per token, and $R$, the frequency of routed layers). They also introduce an "Effective Parameter Count" (EPC) that maps any routed model to an equivalent dense model size, enabling direct performance comparisons. The authors train and evaluate three distinct routing techniques: Sinkhorn-BASE (SMoE), HASH Layers, and Reinforcement Learning (RL-R).

Datasets

The paper mentions training on a "multi-trillion-token compendium of English language text comprising documents from the internet alongside open-source text datasets." Specific datasets are not detailed beyond this general description, but the scale suggests a broad and diverse corpus.

Key Insights and Ideas

  • Two Axes of Scaling: Unlike dense models where parameter count is the primary scaling axis, routed models have two independent axes: the underlying dense model size ($N$) and the number of experts ($E$). Performance improves along both, but with diminishing returns for $E$.
  • Unified Scaling Laws: The authors successfully derive a unified scaling law that accurately describes the performance of various routed architectures, generalizing existing power-law relationships. This law is expressed in terms of both $(N, E)$ and $(F, B)$, demonstrating its broad applicability.
  • Effective Parameter Count (EPC): The concept of EPC is a powerful tool. It allows researchers to equate the performance of a routed model with a specific dense model size, providing a single metric for comparing models with different architectures and expert counts.
  • Diminishing Returns and Saturation: The scaling laws reveal that while increasing the number of experts ($E$) generally improves performance, these gains saturate. This is crucial for efficient model design, preventing unnecessary increases in $E$.
  • Architectural Details Matter Less Than Expected: The study shows that architectural choices like the number of experts per token ($K$) and routing frequency ($R$) have a less significant impact on the overall scaling behavior than $N$ and $E$ when using the $(F, B)$ formulation.

Key Results

  • S-BASE Outperforms: Across various scales and techniques, the Sinkhorn-BASE (S-BASE) routing method consistently achieved the best performance, followed by RL-R, and then HASH.
  • Routing Benefits: All routing techniques showed performance improvements compared to dense models, especially at smaller scales.
  • Scaling Laws Fit Well: The proposed scaling laws accurately predict model performance across five orders of magnitude of size, with low RMSLE values indicating a good fit.
  • Effective Parameter Count: The EPC analysis shows that a routed model with $N=5$M parameters and $E=128$ experts can achieve the same performance as a dense model with $N=55$M parameters, representing a significant computational saving.
  • Ncutoff: Routing benefits diminish as the base model size ($N$) increases. The paper identifies an $N_{cutoff}$ beyond which routing no longer improves performance, with S-BASE having the highest $N_{cutoff}$.

Limitations and Future Work

  • Convergence: The authors note that even their largest models did not fully converge within the training budget (130 billion tokens), suggesting that true convergence might require trillions of tokens. This implies that scaling analyses should focus on performance at a fixed token count rather than assuming convergence.
  • Token Count Dependence: Scaling coefficients are shown to be dependent on the number of tokens seen during training, meaning that comparisons must carefully match training durations.
  • Future Research Directions: The authors suggest future work should focus on understanding the terms $c$ and $E_{max}$ in their scaling laws, which represent limits to arbitrary scaling. They also recommend validating new routing techniques across multiple values of $N$ and $E$, as results from single sizes cannot be extrapolated. Further investigation into the optimal routing policy and the impact of architectural variations like $K$ and $R$ is also encouraged.

Summary updated

Abstract

The performance of a language model has been shown to be effectively modeled as a power-law in its parameter count. Here we study the scaling behaviors of Routing Networks: architectures that conditionally use only a subset of their parameters while processing an input. For these models, parameter count and computational requirement form two independent axes along which an increase leads to better performance. In this work we derive and justify scaling laws defined on these two variables which generalize those known for standard language models and describe the performance of a wide range of routing architectures trained via three different techniques. Afterwards we provide two applications of these laws: first deriving an Effective Parameter Count along which all models scale at the same rate, and then using the scaling coefficients to give a quantitative comparison of the three routing techniques considered. Our analysis derives from an extensive evaluation of Routing Networks across five orders of magnitude of size, including models with hundreds of experts and hundreds of billions of parameters.
BibTeX
@inproceedings{clark2022unified,
  author = {Aidan Clark and Diego de Las Casas and Aurelia Guy and Arthur Mensch and Michela Paganini and Jordan Hoffmann and Bogdan Damoc and Blake A. Hechtman and Trevor Cai and Sebastian Borgeaud and George van den Driessche and Eliza Rutherford and Tom Hennigan and Matthew Johnson and Katie Millican and Albin Cassirer and Chris Jones and Elena Buchatskaya and David Budden and Laurent Sifre and Simon Osindero and Oriol Vinyals and Jack W. Rae and Erich Elsen and Koray Kavukcuoglu and Karen Simonyan},
  title = {{Unified Scaling Laws for Routed Language Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=0_aN1qqexX},
}
Figure