TL;DR
Pre-training auxiliary balancing losses force fine-grained Mixture-of-Experts (MoE) models to distribute tokens evenly, leaving their most impactful "key experts" underutilized while wasting compute on redundant ones. Ban&Pick is a training-free, plug-and-play inference strategy that identifies and promotes decisive key experts (Pick) while dynamically pruning low-impact experts based on layer and token sensitivity (Ban). On models like Qwen3 and DeepSeek-V2, it delivers both higher benchmark accuracy and up to a 1.26× inference speedup.
What Problem Does This Paper Address?
Modern fine-grained MoE architectures (such as DeepSeek-V2.5 and Qwen3) employ up to 160 smaller experts per layer, activating a fixed subset (e.g., top-6 or top-8) for each token. While this fine-grained design allows specialized skills to develop independently, current pre-training recipes hamper its full potential:
- Premature and Constrained Routing: Routers stabilize early in pre-training (often in the first 1%–40% of steps), well before expert weights fully mature. Furthermore, auxiliary load-balancing losses actively penalize the model if it routes too many tokens to a single expert, capping activation frequency even if an expert has become uniquely competent at a given domain.
- Fixed-Budget Inefficiency: Always activating a fixed number of experts per token forces the model to select low-impact experts on easy tokens or resilient layers, introducing substantial computational redundancy.
Methods
Ban&Pick resolves this trade-off post-training without any retraining or architectural modifications:
- Pick (Reinforcing Key Experts):
- Candidate Identification: Pinpoints domain-specialized experts whose selection frequency heavily exceeds a uniform baseline on calibration data (e.g., $>6\times$ on Qwen3).
- Key Expert Discovery: Measures the Kullback–Leibler (KL) divergence of top-1000 output logits when each candidate is individually pruned. Only a tiny fraction of domain-specialized experts cause major distribution shifts; these are labeled "key experts."
- Adaptive Enhancement: At inference, if a key expert falls within the router’s top-$2k$ logits (demonstrating token relevance), it is promoted into the active set. When running Pick alone, it replaces the lowest-weight expert (Strategy D); when combined with Ban, it is added directly (Strategy C).
-
Ban (Dynamic Expert Pruning): Dynamically scales the active expert count per token ($K_{i,l} \in [K_{\min}, K_{\text{base}}]$) by combining two normalized sensitivity signals:
-
Layer Sensitivity ($L'_l$): Calibrated offline via the output KL divergence observed when pruning layer $l$ from $K_{\text{base}}$ down to $K_{\min}$. (Early and late layers are sensitive, while middle layers tolerate aggressive pruning).
- Token Sensitivity ($T'_{i,l}$): Evaluated online during inference via the concentration ratio of top-3 to top-8 routing weights. Tokens with dispersed weights are treated as sensitive, while peaked tokens are aggressively pruned.
Datasets
The authors evaluate Ban&Pick across math, general knowledge, and code benchmarks:
- Evaluation: AIME2024 (evaluated over 5 runs), MATH-500, GSM8K (for smaller models), GPQA-Diamond, LiveCodeBench (V1), and HumanEval+.
- Calibration: MathQA (math), ARC-Challenge (general reasoning), MMLU college-computer-science-compilers (code), and C4.
- Note: No new datasets were introduced in this paper.
Key Insights and Ideas
- Specialization is Capped by Balancing Losses: Even for heavily domain-specialized experts, pre-training balancing penalties cap activation frequencies at around ~50% per task domain. The router "locks in" before experts finish learning, leaving mature, highly capable experts under-called.
- Frequency $\neq$ Importance: While dozens of experts may activate frequently for math or code, selectively removing most of them causes virtually zero logit perturbation. Only 2–3 "key experts" per domain drive the vast majority of performance.
- Top-$2k$ Gating Beats Universal Forcing: Forcibly activating key experts on every token hurts accuracy because it pollutes non-relevant tokens (e.g., punctuation, pronouns). Activating them conditionally when they place within the router's top-$2k$ provides a lightweight, natural filter for task-relevant tokens.
Key Results
Experiments evaluated on DeepSeek-V2-Lite-Chat (16B), DeepSeek-v2.5-1210 (236B), Qwen3-30B-A3B, and Qwen3-235B-A22B under vLLM:
- Pick Alone: Yields consistent accuracy boosts across all models, outperforming baselines like Dynamic Routing, TiP, and RICE (e.g., +4.66% on AIME2024 and +3.53% on GPQA-Diamond for Qwen3-30B-A3B).
- Ban Alone: Prunes average active experts from 8 down to ~4.8 on Qwen3 and from 6 to ~4.0 on DeepSeek, delivering a 1.22×–1.27× inference speedup with minimal accuracy loss ($<1.0\%$ across most tasks), substantially outperforming existing dynamic pruning baselines (DES and ODP).
- Combined Ban&Pick: Achieves simultaneous accuracy improvements and acceleration:
- Qwen3-30B-A3B: Accuracy rises on AIME2024 ($80.67\% \to 84.66\%$) and GPQA ($65.66\% \to 68.18\%$) with a 1.25× speedup and a +28.37% token throughput increase.
- Qwen3-235B-A22B: AIME2024 improves from $84.67\% \to 86.67\%$ alongside a 1.26× speedup.
Limitations and Future Work
- Domain Calibration Dependency: Key experts must currently be identified offline using domain-specific calibration datasets.
- Cross-Domain Expert Interference: Activating key experts for multiple domains simultaneously (e.g., Math + Code + General) slightly dilutes individual domain gains compared to single-domain tuning, though it still outperforms baseline models.
- Future Work: The authors recommend developing automated, input-adaptive policies to detect and boost key experts on-the-fly, as well as studying the deeper mechanistic reasons why specific experts become so uniquely critical.
Summary updated