Complete-muE: Optimal Hyperparameter Transfer and Scaling for MoE Models

Hongwu Peng, Ohiremen Dibua, Yuanjun Xiong, Yifan Gong, Jianming Zhang, Yan KangarXiv 2026 · arXiv

TL;DR

Complete-muE is a novel framework that enables zero-shot hyperparameter transfer from a single tuned dense model to any Mixture-of-Experts (MoE) architecture. By elegantly bridging architectural scaling and expert-side token workload changes, it establishes a practical "tune dense once, transfer to all" recipe, unlocking up to 5.5$\times$ training convergence speedups without the prohibitive cost of tuning MoE hyperparameters from scratch.

What Problem Does This Paper Address?

Scaling up neural networks using Mixture-of-Experts (MoE) layers is highly effective because it increases model capacity without a proportional increase in computational cost. However, optimizing these models is notoriously difficult due to hyperparameter tuning. Hyperparameters (like learning rate, weight decay, and initialization) optimized for a dense model do not typically transfer well to an MoE model.

Existing hyperparameter transfer frameworks fall short for MoEs:

  1. Maximal Update Parametrization ($\mu$P) successfully transfers hyperparameters across different model widths, but it assumes a fixed architecture and cannot account for changes in token workloads per expert.
  2. Stochastic Differential Equation (SDE) rules can predict how hyperparameters should change when token batch sizes or training horizons shift, but they assume the model architecture remains completely static.

MoE scaling uniquely couples both challenges. When you alter MoE dimensions—such as the number of total experts ($N$) or activated experts ($a$)—you simultaneously change the layer architecture and the data exposure (tokens per expert) over the course of training.

Methods

The authors introduce Complete-muE, which untangles this coupling using a clever "two-bridge" system to map a standard dense Feed-Forward Network (FFN) to any sparse MoE configuration:

Figure 1: Complete-muE transfers across dense FFN, Dense MoE, and sparse MoE
  • Bridge I (Dense FFN $\leftrightarrow$ Dense MoE): This bridge handles the structural transformation. It maps a dense FFN to a "Dense MoE" (where all experts are active, $a=N$) using active-width $\mu$P combined with a normalized router scale ($r_a = a$) to preserve the magnitude of the forward pass and parameter updates.
  • Bridge II (Dense MoE $\leftrightarrow$ Sparse MoE): This bridge handles the change in sparsity and data workload. When moving from a dense MoE to a sparse MoE, the token batch size per expert and the total expert training duration scale down by the exact same ratio. Remarkably, within the SDE framework, these two effects apply inverse corrections to the learning rate and weight decay, causing the first-order SDE corrections to cancel out completely.

By composing these two bridges, Complete-muE provides a unified set of layer-level rules (adjusting output multipliers, down-projection initializations, and routing scales) that absorb all architectural variations.

Datasets

The framework was validated across two distinct domains:

  • Language Modeling (LM): Pretrained on the C4 dataset using a GPT-NeoX style tokenizer. Downstream performance was evaluated across 13 standard benchmarks, including MMLU, HellaSwag, ARC-Challenge, BoolQ, and SVAMP.
  • Diffusion Models (DF): A latent-diffusion transformer trained on multi-modal generation tasks, including 256P/512P images, 240P key-frames, and 240P 5-second videos.

Key Insights and Ideas

  • The Workload Cancellation Miracle: The core "aha!" moment of the paper is that changing MoE sparsity does not require a new global learning rate or weight decay transfer rule. Because an expert's batch size and training duration scale identically, their respective SDE adjustments neutralize each other.
  • Bounded Residual Drift ($\sigma_0$ shift): While the first-order SDE corrections cancel out, a minor shift in the expert-side signal-to-noise ratio (SNR) remains. This causes a slight, predictable hyperparameter drift. However, empirical sweeps show this drift is small enough that dense-tuned parameters land safely within the optimal low-loss region of the MoE models.
  • Shared Experts are Optional: In hybrid architectures, adding always-active shared experts alongside routed experts is often thought to improve performance. The authors find that shared experts actually lead to slightly higher loss, meaning their primary value is likely engineering-focused (e.g., overlapping computation and communication) rather than algorithmic.

Key Results

  • Stable Optima Across Sweeps: Extensive hyperparameter sweeps (learning rate, weight decay, initialization standard deviation) demonstrate that the optimal tuning zones for dense models and various MoE configurations remain tightly aligned.
  • Massive Convergence Speedups: When scaling up to larger models (0.62B active / 6.29B total parameters), the MoE models trained with transferred hyperparameters achieved massive training speedups over their dense counterparts to reach the same loss: 4.5$\times$ speedup on 240P video diffusion and 5.3$\times$ to 5.5$\times$ speedup on language models.
Hyperparameter sweeps showing stable optimal loss regions across different numbers of activated experts.
  • Downstream Gains: On the 100k-step language modeling evaluation, the MoE variants consistently outperformed the dense baseline, raising the average downstream benchmark score from 44.3% (dense) to as high as 50.6% (non-grouped MoE).

Limitations and Future Work

  • Approximate Invariance: Because Bridge II relies on an approximate SDE framework (due to the residual $\sigma_0$ SNR shift), the transfer is near-optimal but not perfectly invariant. Extremely sensitive training setups might still experience minor performance drift.
  • Routing Mechanism Dependency: The mathematical exactness of the active-width matching relies on token-choice (top-$k$) routing, which guarantees a deterministic active width per token. The authors note that expert-choice routing, which enforces fixed token quotas per expert, introduces stochastic token assignments per step and may require further theoretical extensions.

Summary updated

Abstract

We propose Complete-muE, a framework which targets hyperparameter transfer across dense FFN and any Mixture-of-Experts (MoE) setups in transformer blocks. Existing tools such as $μ$P (requires fixed architectue) or SDE (requires fixed per-step token count) cannot directly solve the hyperparameter transfer problem in MoE setups because Dense to MoE transfer or MoE total experts scaling changes both architecture and tokens per expert. Complete-muE solves this challenge with a two-bridge system: Bridge~I maps between dense FFN and Dense MoE by active-width $μ$P with a normalized router scale. Bridge~II maps between Dense MoE and sparse MoE by activated-expert scaling, where the first-order SDE LR/WD correction cancels while a bounded residual $σ_0$ shift remains. The resulting transfer rule, which we term as Complete muE, covers changes in activated experts, total capacity, granularity, and shared/group-balanced hybrids for MoE models as well as network width/depth, batch size, and duration changes for general Transformer models. Extensive language model and diffusion model pretraining experiments confirm that complete-muE yields relatively stable hyperparameter optima across model architectures and parameter counts -- with only minor drift consistent with the non-strict SDE behavior of Bridge~II. In practice this drift is small enough that hyperparameters tuned on a single dense reference transfer near-optimally to all MoE configurations -- \emph{tune dense once, transfer to all} is the practical recipe at the core of Complete-muE. This enables MoE models to achieve accelerated convergence speedup over dense models when scaling model capacity without costly hyperparameter search.
BibTeX
@misc{peng2026complete,
  author = {Hongwu Peng and Ohiremen Dibua and Yuanjun Xiong and Yifan Gong and Jianming Zhang and Yan Kang},
  title = {{Complete-muE: Optimal Hyperparameter Transfer and Scaling for MoE Models}},
  year = {2026},
  eprint = {2605.23893},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.23893},
}

Loading related content...

Figure