Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts

Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, Damai DaiSubmission to ICLR 2025 · Rejected

TL;DR

Standard Mixture-of-Experts (MoE) architectures rely on auxiliary losses to prevent expert starvation, but these extra loss terms inject conflicting gradients that harm language model performance. This paper introduces Loss-Free Balancing, an approach that controls routing purely through dynamically adjusted, expert-wise gating biases based on past load feedback. By eliminating auxiliary loss entirely, it avoids gradient interference and future token leakage while achieving lower perplexity and nearly ideal load balance.

What Problem Does This Paper Address?

In sparse MoE models, each token is routed to only a subset ($K$) of total ($N$) feed-forward experts. Without intervention, MoE routers suffer from routing collapse (where a handful of experts process almost all tokens while others starve) and severe load imbalance, which creates GPU memory and compute bottlenecks under expert parallelism.

Historically, models have relied on an auxiliary load-balancing loss. However, this introduces an unavoidable dilemma:

  • A weak auxiliary loss fails to prevent load imbalance.
  • A strong auxiliary loss enforces balance but generates "interference gradients" that directly compete with the primary language modeling objective, degrading final performance.

Alternative approaches like Expert Choice (EC)—where experts select top tokens instead of tokens selecting experts—guarantee perfect balance but violate the causal constraint of autoregressive language models, leaking information from future tokens into current predictions.

Methods

The authors propose Loss-Free Balancing, shifting load balancing from an optimization loss to a closed-loop feedback control mechanism:

  1. Biased Top-$K$ Routing: An expert-specific bias $b_i$ is added to each expert's routing score $s_{i,t}$ strictly during top-$K$ selection: $$\text{Selected if } s_{i,t} + b_i \in \text{TopK}(\{s_{j,t} + b_j\}_{j=1}^N, K)$$ Crucially, this bias is only used for discrete routing decisions; the actual softmax or sigmoid weighting applied to the expert's output remains unbiased ($s_{i,t}$).

  2. Dynamic Bias Feedback Update: After each training step, the framework tallies the token count $c_i$ assigned to each expert across the batch against the ideal average $\overline{c}_i$. The load violation error is $e_i = \overline{c}_i - c_i$. The bias is updated via: $$b_i \leftarrow b_i + u \cdot \text{sign}(e_i)$$ Overloaded experts see their bias decrease, while underloaded experts see it increase.

  3. Causality Preservation: Biases are updated exclusively using historical batch counts, ensuring no future token information within the current sequence leaks backward.

Datasets

  • Training Corpus: A proprietary multilingual dataset developed by DeepSeek-AI, encompassing web text, mathematics, source code, and published literature.
  • Tokenizer: Byte-pair encoding (BPE) trained via Hugging Face Tokenizers with a 32,000-token vocabulary.
  • Evaluation Set: A held-out validation split of approximately 70 million tokens reserved from the training corpus to measure validation perplexity and load violation.

Key Insights and Ideas

  • Decoupling routing distribution from gradients: Traditional auxiliary losses force the routing network's weights to optimize simultaneously for expert competence and artificial uniform distribution. Adjusting an external bias vector breaks this tension, allowing the router weights to follow the language modeling objective cleanly.
  • Quantifying and exposing "Future Token Leakage": The authors prove theoretically that Expert Choice (EC) routing can leak more than $K \log_2\left(\frac{1-R}{R}\right)$ bits per token (over 50 bits/token for a 9-layer MoE with 16 experts). They empirically demonstrate that reducing chunk sizes in EC causes artificial, deceptive drops in training loss (~10%), which vanish when tokens are shuffled across chunks.
  • Natural synergy with Expert Parallelism (EP): As the computation batch size scales with data and expert parallel dimensions, the per-step batch distribution approaches the global distribution, causing load violation to drop significantly—whereas auxiliary loss methods plateau.

Key Results

The authors evaluated models using the fine-grained DeepSeekMoE architecture using sigmoid gating, measuring balance via MaxVio (Maximal Violation from ideal load):

Model Size Tokens Strategy Validation PPL MaxVio (Global)
1B 100B Loss-Controlled ($\alpha = 0.001$) 9.56 0.72
1B 100B Loss-Free (Ours) 9.50 0.04
3B 200B Loss-Controlled ($\alpha = 0.001$) 7.97 0.52
3B 200B Loss-Free (Ours) 7.92 0.04
  • MaxVio Reduction: Loss-Free Balancing achieved a global maximum load violation of just 0.04 (an order of magnitude lower than auxiliary loss baselines).
  • Ablations on Update Rules: Additive bias with a sign step ($u = 0.001$) outperformed proportional updates ($u \cdot e_i$) and multiplicative scaling ($s_{i,t} \cdot b_i$).
  • Softmax Compatibility: When adapted to standard softmax gating, Loss-Free Balancing likewise surpassed the baseline (PPL 9.599 vs. 9.604; MaxVio 0.027 vs. 0.937).

Limitations and Future Work

  • Hyperparameter Sensitivity: The update rate $u$ requires calibration: setting it too small ($u = 0.0001$) causes sluggish initial balancing, while setting it too high ($u = 0.01$) induces late-stage routing oscillations.
  • Scale of Evaluation: Experiments were conducted on 1B and 3B parameter models trained on up to 200B tokens; validation on frontier-scale models (>100B parameters) and multi-trillion token regimes was not covered in this paper.
  • Gating Sensitivity: Under softmax gating, the mutual normalization across all experts makes the routing gaps fragile, requiring a proportional update variant ($b_i + u \cdot e_i$) rather than the simpler sign-based update used for sigmoid gates.

Summary updated

Abstract

For Mixture-of-Experts (MoE) models, an unbalanced expert load will lead to routing collapse or increased computational overhead. Existing methods commonly employ an auxiliary loss to encourage load balance, but a large auxiliary loss will introduce non-negligible interference gradients into training and thus impair the model performance. In order to control load balance while not producing undesired gradients during training, we propose **Loss-Free Balancing**, a new load balancing strategy that operates without auxiliary losses. To be specific, before the top-K routing decision, Loss-Free Balancing will first apply an expert-wise bias to the routing scores of each expert. By dynamically updating the bias of each expert according to its recent load, Loss-Free Balancing can consistently maintain a balanced distribution of expert load. In addition, since Loss-Free Balancing does not produce any interference gradients, it also elevates the upper bound of model performance gained from MoE training. We validate the performance of Loss-Free Balancing on MoE models with up to 3B parameters trained on up to 200B tokens. Experimental results show that Loss-Free Balancing achieves both better performance and better load balance compared with traditional auxiliary-loss-controlled load balancing strategies.
BibTeX
@misc{wang2024auxiliary,
  author = {Lean Wang and Huazuo Gao and Chenggang Zhao and Xu Sun and Damai Dai},
  title = {{Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts}},
  year = {2024},
  eprint = {2408.15664},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.15664},
}

Loading related content...

Figure