Ban&Pick: Ehancing Performance and Efficiency of MoE-LLMs via Smarter Routing

Yuanteng Chen, Peisong Wang, Yuantian Shao, Nanxin Zeng, Chang Xu, Jian ChengSubmission to ICLR 2026 · Rejected

TL;DR

Pre-training auxiliary balancing losses force fine-grained Mixture-of-Experts (MoE) models to distribute tokens evenly, leaving their most impactful "key experts" underutilized while wasting compute on redundant ones. Ban&Pick is a training-free, plug-and-play inference strategy that identifies and promotes decisive key experts (Pick) while dynamically pruning low-impact experts based on layer and token sensitivity (Ban). On models like Qwen3 and DeepSeek-V2, it delivers both higher benchmark accuracy and up to a 1.26× inference speedup.

What Problem Does This Paper Address?

Modern fine-grained MoE architectures (such as DeepSeek-V2.5 and Qwen3) employ up to 160 smaller experts per layer, activating a fixed subset (e.g., top-6 or top-8) for each token. While this fine-grained design allows specialized skills to develop independently, current pre-training recipes hamper its full potential:

  1. Premature and Constrained Routing: Routers stabilize early in pre-training (often in the first 1%–40% of steps), well before expert weights fully mature. Furthermore, auxiliary load-balancing losses actively penalize the model if it routes too many tokens to a single expert, capping activation frequency even if an expert has become uniquely competent at a given domain.
  2. Fixed-Budget Inefficiency: Always activating a fixed number of experts per token forces the model to select low-impact experts on easy tokens or resilient layers, introducing substantial computational redundancy.

Methods

Ban&Pick resolves this trade-off post-training without any retraining or architectural modifications:

  • Pick (Reinforcing Key Experts):
  • Candidate Identification: Pinpoints domain-specialized experts whose selection frequency heavily exceeds a uniform baseline on calibration data (e.g., $>6\times$ on Qwen3).
  • Key Expert Discovery: Measures the Kullback–Leibler (KL) divergence of top-1000 output logits when each candidate is individually pruned. Only a tiny fraction of domain-specialized experts cause major distribution shifts; these are labeled "key experts."
  • Adaptive Enhancement: At inference, if a key expert falls within the router’s top-$2k$ logits (demonstrating token relevance), it is promoted into the active set. When running Pick alone, it replaces the lowest-weight expert (Strategy D); when combined with Ban, it is added directly (Strategy C).
  • Ban (Dynamic Expert Pruning): Dynamically scales the active expert count per token ($K_{i,l} \in [K_{\min}, K_{\text{base}}]$) by combining two normalized sensitivity signals:

  • Layer Sensitivity ($L'_l$): Calibrated offline via the output KL divergence observed when pruning layer $l$ from $K_{\text{base}}$ down to $K_{\min}$. (Early and late layers are sensitive, while middle layers tolerate aggressive pruning).

  • Token Sensitivity ($T'_{i,l}$): Evaluated online during inference via the concentration ratio of top-3 to top-8 routing weights. Tokens with dispersed weights are treated as sensitive, while peaked tokens are aggressively pruned.

Datasets

The authors evaluate Ban&Pick across math, general knowledge, and code benchmarks:

  • Evaluation: AIME2024 (evaluated over 5 runs), MATH-500, GSM8K (for smaller models), GPQA-Diamond, LiveCodeBench (V1), and HumanEval+.
  • Calibration: MathQA (math), ARC-Challenge (general reasoning), MMLU college-computer-science-compilers (code), and C4.
  • Note: No new datasets were introduced in this paper.

Key Insights and Ideas

  • Specialization is Capped by Balancing Losses: Even for heavily domain-specialized experts, pre-training balancing penalties cap activation frequencies at around ~50% per task domain. The router "locks in" before experts finish learning, leaving mature, highly capable experts under-called.
  • Frequency $\neq$ Importance: While dozens of experts may activate frequently for math or code, selectively removing most of them causes virtually zero logit perturbation. Only 2–3 "key experts" per domain drive the vast majority of performance.
  • Top-$2k$ Gating Beats Universal Forcing: Forcibly activating key experts on every token hurts accuracy because it pollutes non-relevant tokens (e.g., punctuation, pronouns). Activating them conditionally when they place within the router's top-$2k$ provides a lightweight, natural filter for task-relevant tokens.

Key Results

Experiments evaluated on DeepSeek-V2-Lite-Chat (16B), DeepSeek-v2.5-1210 (236B), Qwen3-30B-A3B, and Qwen3-235B-A22B under vLLM:

  • Pick Alone: Yields consistent accuracy boosts across all models, outperforming baselines like Dynamic Routing, TiP, and RICE (e.g., +4.66% on AIME2024 and +3.53% on GPQA-Diamond for Qwen3-30B-A3B).
  • Ban Alone: Prunes average active experts from 8 down to ~4.8 on Qwen3 and from 6 to ~4.0 on DeepSeek, delivering a 1.22×–1.27× inference speedup with minimal accuracy loss ($<1.0\%$ across most tasks), substantially outperforming existing dynamic pruning baselines (DES and ODP).
  • Combined Ban&Pick: Achieves simultaneous accuracy improvements and acceleration:
  • Qwen3-30B-A3B: Accuracy rises on AIME2024 ($80.67\% \to 84.66\%$) and GPQA ($65.66\% \to 68.18\%$) with a 1.25× speedup and a +28.37% token throughput increase.
  • Qwen3-235B-A22B: AIME2024 improves from $84.67\% \to 86.67\%$ alongside a 1.26× speedup.

Limitations and Future Work

  • Domain Calibration Dependency: Key experts must currently be identified offline using domain-specific calibration datasets.
  • Cross-Domain Expert Interference: Activating key experts for multiple domains simultaneously (e.g., Math + Code + General) slightly dilutes individual domain gains compared to single-domain tuning, though it still outperforms baseline models.
  • Future Work: The authors recommend developing automated, input-adaptive policies to detect and boost key experts on-the-fly, as well as studying the deeper mechanistic reasons why specific experts become so uniquely critical.

Summary updated

Abstract

Sparse Mixture-of-Experts (MoE) has become a key architecture for scaling large language models (LLMs) efficiently. Recent fine‑grained MoE designs introduce hundreds of experts per layer, with multiple experts activated per token, enabling stronger specialization. However, during pre‑training, routers are optimized mainly for stability and robustness: they converge prematurely and enforce balanced usage, limiting the full potential of model performance and efficiency. In this work, we uncover two overlooked issues: (i) a few highly influential experts are underutilized due to premature and balanced routing decisions; and (ii) enforcing a fixed number of active experts per token introduces substantial redundancy. Instead of retraining models or redesigning MoE architectures, we introduce Ban&Pick, a post-training, plug-and-play strategy for smarter MoE routing. Pick discovers and reinforces key experts—a small group with outsized impact on performance—leading to notable accuracy gains across domains. Ban complements this by dynamically pruning redundant experts based on layer and token sensitivity, delivering faster inference with minimal accuracy loss. Experiments on fine-grained MoE-LLMs (DeepSeek, Qwen3) across math, code, and general reasoning benchmarks demonstrate that Ban\&Pick delivers free performance gains and inference acceleration without retraining or architectural changes. For instance, on Qwen3-30B-A3B, it improves accuracy from 80.67 to 84.66 on AIME2024 and from 65.66 to 68.18 on GPQA-Diamond, while accelerating inference by 1.25× under the vLLM.
BibTeX
@misc{chen2025ban,
  author = {Yuanteng Chen and Peisong Wang and Yuantian Shao and Nanxin Zeng and Chang Xu and Jian Cheng},
  title = {{Ban\&Pick: Ehancing Performance and Efficiency of MoE-LLMs via Smarter Routing}},
  year = {2025},
  eprint = {2509.06346},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.06346},
}
Figure