Unified Scaling Laws for Routed Language Models

Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake A. Hechtman, Trevor Cai, Sebastian Borgeaud, George van den Driessche, Eliza Rutherford, Tom Hennigan, Matthew J. Johnson, Albin Cassirer, Chris Jones, Elena Buchatskaya, David Budden, Laurent Sifre, Simon Osindero, Oriol Vinyals, Marc'Aurelio Ranzato, Jack W. Rae, Erich Elsen, Koray Kavukcuoglu, Karen Simonyan

Overview

This paper introduces unified scaling laws for "routed" language models, such as Mixture-of-Experts (MoE), which decouple a model's total parameter count from its computational cost. By analyzing models scaled up to 200 billion parameters, the authors demonstrate that performance follows a predictable bilinear power law. This framework allows researchers to accurately forecast the performance benefits of adding more "experts" versus increasing the base model size.

The Challenge of Scaling Routed Networks

Standard "dense" language models follow well-established scaling laws: as the number of parameters increases, the validation loss drops predictably. However, "Routing Networks"—architectures that activate only a subset of their parameters for any given input—break this simple relationship.

In routed models, the total parameter count and the actual compute utilized per token are independent. This bifurcation makes it difficult to predict how these models will scale or to fairly compare different routing strategies. The authors set out to establish a mathematical framework that unifies these two axes of size (compute and parameters) into a single, predictable law.

Routing Strategies and the Bilinear Scaling Law

The researchers evaluated three distinct routing techniques across a wide range of scales:

  1. Sinkhorn-BASE (S-BASE): An advanced MoE method utilizing the Sinkhorn algorithm to ensure tokens are distributed evenly among experts, preventing some experts from being overworked while others remain idle.
  2. HASH Layers: A simple, non-parametric approach where tokens are assigned to experts based on a fixed hash of their token ID.
  3. Reinforcement Learning (RL-R): A method that treats expert selection as a policy learned via reinforcement learning (specifically, REINFORCE with a learned baseline).

To model these architectures, the authors proposed a bilinear scaling law where the log-loss is a function of both the log of the "dense" model size (compute per token) and the log of the number of experts. Furthermore, they introduced the Effective Parameter Count (EPC), a metric that translates the performance of a sparse routed model into the size of an equivalent dense model.

Experimental Setup

The models were trained on a massive, multi-trillion-token compendium of English text, comprising web documents and open-source datasets. To test generalization, the authors evaluated the models on several zero-shot benchmarks, including LAMBADA, The Pile, Curation Corpus, WikiText-103, and C4.

Key Insights and Results

  • Routing Consistently Helps: Across all tested model sizes and techniques, adding experts improved performance without significantly increasing inference compute.
  • Diminishing Returns: The performance benefit of adding experts decreases as the base model gets larger. This phenomenon is captured by a positive "interaction term" in their scaling law.
  • RL is Competitive: Often considered difficult to stabilize, reinforcement learning for routing proved surprisingly competitive with state-of-the-art MoE techniques like S-BASE.
  • Predictable Scaling: The proposed bilinear scaling law accurately describes performance across five orders of magnitude in model size. S-BASE emerged as the most robust and scalable method, consistently outperforming Hash and RL-R at larger scales.

Using the EPC metric, the authors demonstrated that a small routed model can match the performance of a dense model that requires over ten times the compute. For example, a 5M parameter base model with 128 experts performs as well as a 55M parameter dense model.

Unified scaling laws demonstrating how validation loss decreases with expert count and base model size, alongside the unified model scaling curve.

Limitations and Future Directions

The scaling laws predict a theoretical "cutoff" size ($N_{cutoff}$) for each method. Beyond this base model size, adding more experts is predicted to stop providing any benefit. For S-BASE, this limit is estimated to be around 900 billion parameters. However, the authors note that these $N_{cutoff}$ values are extrapolations and are highly sensitive to small changes in the fitted parameters.

Additionally, the models in this study were trained on a fixed number of tokens (130 billion), which is not necessarily the compute-optimal amount of data for every size. Future work is needed to observe how these laws adapt when models are trained to optimality (the "Chinchilla" regime). A major goal for future research is discovering routing methods where the interaction term is closer to zero, potentially allowing the benefits of sparsity to persist even as base models scale into the trillions of parameters.

Summary updated

Abstract

The performance of a language model has been shown to be effectively modeled as a power-law in its parameter count. Here we study the scaling behaviors of Routing Networks: architectures that condi...
BibTeX
@inproceedings{clark2022unified,
  author = {Aidan Clark and Diego de Las Casas and Aurelia Guy and Arthur Mensch and Michela Paganini and Jordan Hoffmann and Bogdan Damoc and Blake A. Hechtman and Trevor Cai and Sebastian Borgeaud and George van den Driessche and Eliza Rutherford and Tom Hennigan and Matthew J. Johnson and Albin Cassirer and Chris Jones and Elena Buchatskaya and David Budden and Laurent Sifre and Simon Osindero and Oriol Vinyals and Marc'Aurelio Ranzato and Jack W. Rae and Erich Elsen and Koray Kavukcuoglu and Karen Simonyan},
  title = {{Unified Scaling Laws for Routed Language Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=0_aN1qqexX},
}
Figure