TL;DR
LASER is a plug-and-play, inference-time routing algorithm that smooths load imbalance across Mixture-of-Experts (MoE) models without requiring any retraining or fine-tuning. By dynamically broadening expert candidate pools when gate scores are diffuse and sticking to top-$k$ experts when scores are sharply peaked, LASER cuts expert load imbalance by up to $1.92\times$ while keeping downstream accuracy within 2% of baseline models.
What Problem Does This Paper Address?
Mixture-of-Experts (MoE) architectures achieve high parameter capacity with low per-token compute by routing each token to a sparse subset of $k$ experts. However, standard top-$k$ routing creates significant inference bottlenecks. Because tokens are routed strictly to the highest-scoring experts, a small subset of "hot" experts becomes overloaded while other "cold" experts sit idle.
In distributed serving, a decoding step completes only when the slowest, most loaded GPU finishes processing its assigned tokens. Training-time balancing tricks (such as auxiliary losses or gate bias adjustments) fail to eliminate load spikes during inference, and modifying weights requires expensive retraining. The core challenge is: Can we dynamically balance inference load at runtime without degrading response quality or altering pre-trained weights?
Methods
LASER operates immediately after the gating layer during the forward pass by inspecting the per-token gate score distribution:
- Assessing Distribution Skew: The algorithm computes the top-$k$ probability mass $M_k = \sum_{i=1}^k s_{(i)}$. If $M_k \ge \varepsilon_{\text{high}}$, one or a few experts clearly dominate; LASER immediately routes to the vanilla top-$k$ experts to safeguard output quality.
- Dynamic Candidate Pool Construction: When scores are diffuse ($M_k < \varepsilon_{\text{high}}$), LASER establishes a relative threshold $t = t_{\text{fix}} \cdot s_{(1)}$ based on the top score. All experts with $s_i \ge t$ (plus the top-$k$ experts) form an expanded candidate pool $\mathcal{T}$.
- Trimming and Load-Aware Assignment: To bound selection overhead, the pool is trimmed to a working size $c$ ($k \le c \le |\mathcal{T}|$). LASER sorts candidates by current load $L_e$ in ascending order (breaking ties by score) and assigns the token to the $k$ least-loaded experts.
Because gating dynamics vary across network depth, LASER supports grouping layers into bands (early, middle, final) with band-specific thresholds calibrated from prefill traces.
Datasets
The authors evaluated LASER across four standard NLP and reasoning benchmarks:
- GSM8K (grade-school math word problems; 256 samples)
- MMLU (multi-subject factual and conceptual multiple-choice; 512 samples across 57 subjects)
- ARC-Easy and ARC-Challenge (elementary and advanced science question answering; 512 samples each)
- (Empirical analysis only) Wiki (WikiText) was used alongside GSM8K and MMLU to analyze layer-wise entropy and probability mass.
No new benchmarks were created; all datasets and pre-trained models are publicly available.
Key Insights and Ideas
- Layer-wise routing profiles vary predictably: Early and late layers exhibit sharp, low-entropy gate score distributions ("single-head" regimes) where top experts dominate. In contrast, middle layers are much flatter and higher-entropy ("smooth" or "plateau" regimes).
- Entropy creates balancing headroom: In middle layers, multiple experts demonstrate comparable relevance for a given token. Routing to the least-loaded expert among viable candidates balances hardware utilization without sacrificing model expressiveness.
- Pure load balancing fails: Forcing uniform token distribution without score constraints ("load-only" routing) catastrophically degrades task accuracy, underscoring that candidate expansion must be strictly bounded by score thresholds.
Key Results
- Evaluated Models: Evaluated on Mixtral-8×7B ($k=2$, 8 experts) and DeepSeek-MoE-16b-chat ($k=6$, 64 fine-grained experts).
- Substantial Load Reduction: LASER reduced peak expert-load imbalance by up to $1.92\times$ ($\approx 48\%$). On aggregate imbalance ($I_{\text{agg}}$), Mixtral saw up to a $1.63\times$ reduction on GSM8K, while DeepSeek achieved $\approx \mathbf{1.4\times}$ reductions on MMLU and ARC-Challenge.
- Preserved Accuracy: Downstream task accuracy remained within $0.02$ absolute ($\le 2\%$) of vanilla top-$k$ routing across all benchmarks.
- Middle Layer Wins: As predicted by gate score entropy, maximum load violation dropped most dramatically in the middle transformer layers.
Limitations and Future Work
- Threshold Calibration: Current thresholds ($\varepsilon_{\text{high}}$ and $t_{\text{fix}}$) are tuned per dataset using prefill profiles rather than adjusted dynamically online. Automated tuning or runtime self-adaptation remains an open problem.
- Hardware-Level Evaluation: Experiments reported expert-level imbalance using single-GPU execution rather than wall-clock distributed throughput across large multi-node clusters with expert parallelism.
- Broader Routing Objectives: The candidate selection ranking currently sorts only by expert load. Future iterations could integrate communication locality (e.g., favoring co-located intra-node experts) or GPU memory headroom constraints.
Summary updated