Towards Principled Design of Mixture-of-Experts Language Models under Memory and Inference Constraints

Seng Pei Liew, Kenta Shinzato, Yuyang Dong

TL;DR

While Mixture-of-Experts (MoE) models are usually defined by their total and active parameter counts, these metrics don't tell the whole story. This paper reveals a hidden "expert penalty"—where having too many experts forces a reduction in the model's core dimensions—and provides a new scaling law and design principle: to optimize performance, you should maximize total parameters while minimizing both expert sparsity and the total number of experts.

What Problem Does This Paper Address?

Mixture-of-Experts (MoE) models have become the standard for scaling large language models (LLMs) efficiently. By only activating a fraction of the model (the "active parameters") for any given token, they provide the performance of a massive model with the inference cost of a much smaller one.

However, model designers face a massive search space. If you have a budget for 100B total parameters and 20B active parameters, how do you choose the number of layers, the hidden width, the number of experts ($n_{exp}$), and how many to activate ($n_{topk}$)? Current literature often treats the "sparsity ratio" ($n_{exp}/n_{topk}$) as the primary factor, but the authors found that different architectures with the same sparsity and parameter counts perform differently. This paper aims to resolve that architectural ambiguity.

Methods

The researchers conducted a systematic study using a Qwen3-like architecture. Their approach involved two main phases:

  1. Ablation Studies: They fixed the total and active parameter budgets and varied individual components like granularity ($g$, the ratio of hidden dimension to expert dimension) and the width-to-depth ratio ($\gamma = d/l$) to find optimal ranges.
  2. Scaling Law Derivation: They trained a wide range of models (from 30M to 3B parameters) across various configurations and used linear regression on a log-log scale to fit a new scaling law that accounts for $N_{total}$, $n_{exp}$, and $n_{topk}$.
  3. Optimization Routine: They developed an iterative algorithm (Algorithm 1) that takes memory and inference constraints as input and outputs the optimal $l, d, n_{exp},$ and $n_{topk}$.

Datasets

The models were trained on the FineWeb-Edu dataset, a high-quality subset of web data. Performance was evaluated using loss on a held-out validation set. Training durations varied from 9B to 50B tokens to validate the scaling laws across different data regimes.

Key Insights and Ideas

  • The $n_{exp}$ Penalty: This is the paper's most critical finding. If you keep the total parameter count ($N_{total}$) fixed but increase the number of experts ($n_{exp}$), you are forced to shrink the "core" dimensions of the model (depth and width) to stay within the memory budget. This reduction in the core model leads to a net decrease in active compute capacity per token, which hurts performance.
  • Sparsity vs. Scale: Total parameters ($N_{total}$) are a much stronger predictor of performance than active parameters ($N_{active}$).
  • Granularity Sweet Spot: The ratio of the hidden dimension to the expert hidden dimension ($g$) should ideally be between 4 and 8. Going higher yields diminishing returns.
  • Width-to-Depth Balance: There is a "sweet spot" for the width-to-depth ratio ($\gamma$) between 32 and 64. Models that are too deep and skinny or too shallow and wide perform worse under fixed parameter budgets.

Key Results

The authors proposed a new scaling law for MoE loss ($L$): $$L \propto N_{total}^{-0.052} \cdot s^{0.018} \cdot n_{exp}^{0.005}$$ (where $s$ is the sparsity ratio $n_{exp}/n_{topk}$)

  • Validation of the Principle: They compared two configurations with the same sparsity ratio ($s=16$): one with $(n_{exp}=128, n_{topk}=8)$ and another with $(256, 16)$. According to their theory, the 128-expert model should win because it incurs a lower "$n_{exp}$ penalty." Experimental results confirmed this, showing the 128-expert model consistently outperformed the 256-expert model across all token counts.
  • Real-world Application: Using their optimization routine, they analyzed the Qwen3-235B-A22B model. Their framework suggested an optimal configuration of 94 layers (vs. the actual 83) and a slightly different expert setup, potentially offering better performance within the same memory footprint.

Limitations and Future Work

  • Scale: Due to compute constraints, the experiments were limited to models up to 3B parameters. While scaling laws usually hold, testing on 100B+ parameter models is necessary for absolute certainty.
  • Parameter Proxies: The paper uses parameter counts as proxies for memory and latency. In the real world, factors like hardware-specific kernels (e.g., FlashAttention), communication overhead in distributed training, and KV cache size also play significant roles in "optimal" design.
  • Search Breadth: The authors used a greedy search for some hyperparameters ($g$ and $\gamma$) rather than a full grid search across all five dimensions simultaneously.

Summary updated

Abstract

Modern Mixture-of-Experts (MoE) language models are designed based on total parameters (memory footprint) and active parameters (inference cost). However, we find these two factors alone are insufficient to describe an optimal architecture. Through a systematic study, we demonstrate that MoE performance is primarily determined by total parameters ($N_{total}$) and expert sparsity ($s:=n_{exp}/n_{topk}$). Moreover, $n_{exp}$ and $n_{topk}$ do not "cancel out" within the sparsity ratio; instead, a larger total number of experts slightly penalizes performance by forcing a reduction in core model dimensions (depth and width) to meet memory constraints. This motivates a simple principle for MoE design which maximizes $N_{total}$ while minimizing $s$ (maximizing $n_{topk}$) and $n_{exp}$ under the given constraints. Our findings provide a robust framework for resolving architectural ambiguity and guiding MoE design.
BibTeX
@misc{liew2026towards,
  author = {Seng Pei Liew and Kenta Shinzato and Yuyang Dong},
  title = {{Towards Principled Design of Mixture-of-Experts Language Models under Memory and Inference Constraints}},
  year = {2026},
  eprint = {2601.08215},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.08215},
}
Figure