TL;DR
This paper introduces unified scaling laws for "routed" language models, which use only a subset of their parameters per input. These laws show that performance scales with both model size and the number of "experts" (parameter subsets), offering a more nuanced understanding than traditional scaling laws. This work provides a framework for comparing different routing techniques and designing more efficient large language models.
What Problem Does This Paper Address?
For years, the scaling of neural networks, particularly language models, has been observed to follow power-law relationships: as model size (parameter count) increases, performance (typically measured by loss) decreases predictably. This has been a cornerstone for building larger and more capable models. However, a new class of architectures called "Routing Networks" (or Mixture-of-Experts models) complicates this picture. These models use a "router" to select a subset of parameters (experts) for each input, meaning parameter count and computational cost are no longer directly coupled. This disconnect makes traditional scaling laws insufficient. The authors aim to develop new scaling laws that accurately describe these routed models, generalize existing ones, and provide a unified understanding of their performance across different architectures and training methods.
Methods
The core of this paper is the derivation and validation of new scaling laws for routed language models. They propose a bilinear scaling law that considers both the dense model size ($N$) and the number of experts ($E$), and crucially, incorporates a saturating transformation for $E$ to account for diminishing returns. This law is then generalized to use inference compute ($F$) and parameter utilization ratio ($B$) as variables, allowing for a unified scaling law across different routing architectures (varying $K$, the number of experts used per token, and $R$, the frequency of routed layers). They also introduce an "Effective Parameter Count" (EPC) that maps any routed model to an equivalent dense model size, enabling direct performance comparisons. The authors train and evaluate three distinct routing techniques: Sinkhorn-BASE (SMoE), HASH Layers, and Reinforcement Learning (RL-R).
Datasets
The paper mentions training on a "multi-trillion-token compendium of English language text comprising documents from the internet alongside open-source text datasets." Specific datasets are not detailed beyond this general description, but the scale suggests a broad and diverse corpus.
Key Insights and Ideas
- Two Axes of Scaling: Unlike dense models where parameter count is the primary scaling axis, routed models have two independent axes: the underlying dense model size ($N$) and the number of experts ($E$). Performance improves along both, but with diminishing returns for $E$.
- Unified Scaling Laws: The authors successfully derive a unified scaling law that accurately describes the performance of various routed architectures, generalizing existing power-law relationships. This law is expressed in terms of both $(N, E)$ and $(F, B)$, demonstrating its broad applicability.
- Effective Parameter Count (EPC): The concept of EPC is a powerful tool. It allows researchers to equate the performance of a routed model with a specific dense model size, providing a single metric for comparing models with different architectures and expert counts.
- Diminishing Returns and Saturation: The scaling laws reveal that while increasing the number of experts ($E$) generally improves performance, these gains saturate. This is crucial for efficient model design, preventing unnecessary increases in $E$.
- Architectural Details Matter Less Than Expected: The study shows that architectural choices like the number of experts per token ($K$) and routing frequency ($R$) have a less significant impact on the overall scaling behavior than $N$ and $E$ when using the $(F, B)$ formulation.
Key Results
- S-BASE Outperforms: Across various scales and techniques, the Sinkhorn-BASE (S-BASE) routing method consistently achieved the best performance, followed by RL-R, and then HASH.
- Routing Benefits: All routing techniques showed performance improvements compared to dense models, especially at smaller scales.
- Scaling Laws Fit Well: The proposed scaling laws accurately predict model performance across five orders of magnitude of size, with low RMSLE values indicating a good fit.
- Effective Parameter Count: The EPC analysis shows that a routed model with $N=5$M parameters and $E=128$ experts can achieve the same performance as a dense model with $N=55$M parameters, representing a significant computational saving.
- Ncutoff: Routing benefits diminish as the base model size ($N$) increases. The paper identifies an $N_{cutoff}$ beyond which routing no longer improves performance, with S-BASE having the highest $N_{cutoff}$.
Limitations and Future Work
- Convergence: The authors note that even their largest models did not fully converge within the training budget (130 billion tokens), suggesting that true convergence might require trillions of tokens. This implies that scaling analyses should focus on performance at a fixed token count rather than assuming convergence.
- Token Count Dependence: Scaling coefficients are shown to be dependent on the number of tokens seen during training, meaning that comparisons must carefully match training durations.
- Future Research Directions: The authors suggest future work should focus on understanding the terms $c$ and $E_{max}$ in their scaling laws, which represent limits to arbitrary scaling. They also recommend validating new routing techniques across multiple values of $N$ and $E$, as results from single sizes cannot be extrapolated. Further investigation into the optimal routing policy and the impact of architectural variations like $K$ and $R$ is also encouraged.
Summary updated