From Score Distributions to Balance: Plug-and-Play Mixture-of-Experts Routing

Rana Shahout, Colin Cai, Yilun Du, Minlan Yu, Michael MitzenmacherSubmission to ICLR 2026 · Withdrawn

TL;DR

LASER is a plug-and-play, inference-time routing algorithm that smooths load imbalance across Mixture-of-Experts (MoE) models without requiring any retraining or fine-tuning. By dynamically broadening expert candidate pools when gate scores are diffuse and sticking to top-$k$ experts when scores are sharply peaked, LASER cuts expert load imbalance by up to $1.92\times$ while keeping downstream accuracy within 2% of baseline models.

What Problem Does This Paper Address?

Mixture-of-Experts (MoE) architectures achieve high parameter capacity with low per-token compute by routing each token to a sparse subset of $k$ experts. However, standard top-$k$ routing creates significant inference bottlenecks. Because tokens are routed strictly to the highest-scoring experts, a small subset of "hot" experts becomes overloaded while other "cold" experts sit idle.

In distributed serving, a decoding step completes only when the slowest, most loaded GPU finishes processing its assigned tokens. Training-time balancing tricks (such as auxiliary losses or gate bias adjustments) fail to eliminate load spikes during inference, and modifying weights requires expensive retraining. The core challenge is: Can we dynamically balance inference load at runtime without degrading response quality or altering pre-trained weights?

Methods

LASER operates immediately after the gating layer during the forward pass by inspecting the per-token gate score distribution:

  1. Assessing Distribution Skew: The algorithm computes the top-$k$ probability mass $M_k = \sum_{i=1}^k s_{(i)}$. If $M_k \ge \varepsilon_{\text{high}}$, one or a few experts clearly dominate; LASER immediately routes to the vanilla top-$k$ experts to safeguard output quality.
  2. Dynamic Candidate Pool Construction: When scores are diffuse ($M_k < \varepsilon_{\text{high}}$), LASER establishes a relative threshold $t = t_{\text{fix}} \cdot s_{(1)}$ based on the top score. All experts with $s_i \ge t$ (plus the top-$k$ experts) form an expanded candidate pool $\mathcal{T}$.
  3. Trimming and Load-Aware Assignment: To bound selection overhead, the pool is trimmed to a working size $c$ ($k \le c \le |\mathcal{T}|$). LASER sorts candidates by current load $L_e$ in ascending order (breaking ties by score) and assigns the token to the $k$ least-loaded experts.

Because gating dynamics vary across network depth, LASER supports grouping layers into bands (early, middle, final) with band-specific thresholds calibrated from prefill traces.

Datasets

The authors evaluated LASER across four standard NLP and reasoning benchmarks:

  • GSM8K (grade-school math word problems; 256 samples)
  • MMLU (multi-subject factual and conceptual multiple-choice; 512 samples across 57 subjects)
  • ARC-Easy and ARC-Challenge (elementary and advanced science question answering; 512 samples each)
  • (Empirical analysis only) Wiki (WikiText) was used alongside GSM8K and MMLU to analyze layer-wise entropy and probability mass.

No new benchmarks were created; all datasets and pre-trained models are publicly available.

Key Insights and Ideas

  • Layer-wise routing profiles vary predictably: Early and late layers exhibit sharp, low-entropy gate score distributions ("single-head" regimes) where top experts dominate. In contrast, middle layers are much flatter and higher-entropy ("smooth" or "plateau" regimes).
  • Entropy creates balancing headroom: In middle layers, multiple experts demonstrate comparable relevance for a given token. Routing to the least-loaded expert among viable candidates balances hardware utilization without sacrificing model expressiveness.
  • Pure load balancing fails: Forcing uniform token distribution without score constraints ("load-only" routing) catastrophically degrades task accuracy, underscoring that candidate expansion must be strictly bounded by score thresholds.

Key Results

  • Evaluated Models: Evaluated on Mixtral-8×7B ($k=2$, 8 experts) and DeepSeek-MoE-16b-chat ($k=6$, 64 fine-grained experts).
  • Substantial Load Reduction: LASER reduced peak expert-load imbalance by up to $1.92\times$ ($\approx 48\%$). On aggregate imbalance ($I_{\text{agg}}$), Mixtral saw up to a $1.63\times$ reduction on GSM8K, while DeepSeek achieved $\approx \mathbf{1.4\times}$ reductions on MMLU and ARC-Challenge.
  • Preserved Accuracy: Downstream task accuracy remained within $0.02$ absolute ($\le 2\%$) of vanilla top-$k$ routing across all benchmarks.
  • Middle Layer Wins: As predicted by gate score entropy, maximum load violation dropped most dramatically in the middle transformer layers.

Limitations and Future Work

  • Threshold Calibration: Current thresholds ($\varepsilon_{\text{high}}$ and $t_{\text{fix}}$) are tuned per dataset using prefill profiles rather than adjusted dynamically online. Automated tuning or runtime self-adaptation remains an open problem.
  • Hardware-Level Evaluation: Experiments reported expert-level imbalance using single-GPU execution rather than wall-clock distributed throughput across large multi-node clusters with expert parallelism.
  • Broader Routing Objectives: The candidate selection ranking currently sorts only by expert load. Future iterations could integrate communication locality (e.g., favoring co-located intra-node experts) or GPU memory headroom constraints.

Summary updated

Abstract

Mixture-of-Experts (MoE) models can scale parameter capacity by routing each token to a subset of experts through a learned gate function. While conditional routing reduces training costs, it shifts the burden on inference memory: expert parameters and activations consume memory, limiting the number of experts per device. As tokens are routed, some experts become overloaded while others are underutilized. Because experts are mapped to GPUs, this imbalance translates di- rectly into degraded system performance in terms of latency, throughput, and cost. We present LASER, a plug-and-play, inference-time routing algorithm that bal- ances load while preserving accuracy. LASER adapts to the shape of the gate’s score distribution. When scores provide a clear preference, it routes to the strongest experts; when scores are more uniform, it broadens the set of viable ex- perts and routes to the least-loaded among them. Because LASER relies only on gate scores from a trained model, it integrates directly into existing MoE inference pipelines without retraining or finetuning. We evaluate LASER on Mixtral-8×7B and DeepSeek-MoE-16b-chat across four datasets (ARC-Easy, ARC-Challenge, MMLU, and GSM8K). LASER improves load balancing, translating into lower latency and higher throughput, while keeping accuracy changes negligible
BibTeX
@misc{shahout2025from,
  author = {Rana Shahout and Colin Cai and Yilun Du and Minlan Yu and Michael Mitzenmacher},
  title = {{From Score Distributions to Balance: Plug-and-Play Mixture-of-Experts Routing}},
  year = {2025},
  eprint = {2510.03293},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.03293},
}
Figure