Overview
This paper introduces unified scaling laws for "routed" language models, such as Mixture-of-Experts (MoE), which decouple a model's total parameter count from its computational cost. By analyzing models scaled up to 200 billion parameters, the authors demonstrate that performance follows a predictable bilinear power law. This framework allows researchers to accurately forecast the performance benefits of adding more "experts" versus increasing the base model size.
The Challenge of Scaling Routed Networks
Standard "dense" language models follow well-established scaling laws: as the number of parameters increases, the validation loss drops predictably. However, "Routing Networks"—architectures that activate only a subset of their parameters for any given input—break this simple relationship.
In routed models, the total parameter count and the actual compute utilized per token are independent. This bifurcation makes it difficult to predict how these models will scale or to fairly compare different routing strategies. The authors set out to establish a mathematical framework that unifies these two axes of size (compute and parameters) into a single, predictable law.
Routing Strategies and the Bilinear Scaling Law
The researchers evaluated three distinct routing techniques across a wide range of scales:
- Sinkhorn-BASE (S-BASE): An advanced MoE method utilizing the Sinkhorn algorithm to ensure tokens are distributed evenly among experts, preventing some experts from being overworked while others remain idle.
- HASH Layers: A simple, non-parametric approach where tokens are assigned to experts based on a fixed hash of their token ID.
- Reinforcement Learning (RL-R): A method that treats expert selection as a policy learned via reinforcement learning (specifically, REINFORCE with a learned baseline).
To model these architectures, the authors proposed a bilinear scaling law where the log-loss is a function of both the log of the "dense" model size (compute per token) and the log of the number of experts. Furthermore, they introduced the Effective Parameter Count (EPC), a metric that translates the performance of a sparse routed model into the size of an equivalent dense model.
Experimental Setup
The models were trained on a massive, multi-trillion-token compendium of English text, comprising web documents and open-source datasets. To test generalization, the authors evaluated the models on several zero-shot benchmarks, including LAMBADA, The Pile, Curation Corpus, WikiText-103, and C4.
Key Insights and Results
- Routing Consistently Helps: Across all tested model sizes and techniques, adding experts improved performance without significantly increasing inference compute.
- Diminishing Returns: The performance benefit of adding experts decreases as the base model gets larger. This phenomenon is captured by a positive "interaction term" in their scaling law.
- RL is Competitive: Often considered difficult to stabilize, reinforcement learning for routing proved surprisingly competitive with state-of-the-art MoE techniques like S-BASE.
- Predictable Scaling: The proposed bilinear scaling law accurately describes performance across five orders of magnitude in model size. S-BASE emerged as the most robust and scalable method, consistently outperforming Hash and RL-R at larger scales.
Using the EPC metric, the authors demonstrated that a small routed model can match the performance of a dense model that requires over ten times the compute. For example, a 5M parameter base model with 128 experts performs as well as a 55M parameter dense model.
Limitations and Future Directions
The scaling laws predict a theoretical "cutoff" size ($N_{cutoff}$) for each method. Beyond this base model size, adding more experts is predicted to stop providing any benefit. For S-BASE, this limit is estimated to be around 900 billion parameters. However, the authors note that these $N_{cutoff}$ values are extrapolations and are highly sensitive to small changes in the fitted parameters.
Additionally, the models in this study were trained on a fixed number of tokens (130 billion), which is not necessarily the compute-optimal amount of data for every size. Future work is needed to observe how these laws adapt when models are trained to optimality (the "Chinchilla" regime). A major goal for future research is discovering routing methods where the interaction term is closer to zero, potentially allowing the benefits of sparsity to persist even as base models scale into the trillions of parameters.
Summary updated