TL;DR
When deactivating experts in Mixture-of-Experts (MoE) models to save on deployment costs, the mathematical rule used to weight the remaining experts must strictly match the rule used during the model's original training. Mismatching these rules—such as renormalizing weights in a model trained on raw probabilities—can lead to catastrophic performance degradation, increasing perplexity by as much as 28,000x.
What Problem Does This Paper Address?
Mixture-of-Experts (MoE) architectures like DeepSeek-V2 and OLMoE are highly efficient because they only activate a fraction of their parameters for any given token. However, the full "expert bank" still occupies massive amounts of VRAM. To reduce these deployment costs, researchers often use expert deactivation (pruning or offloading).
While existing literature focuses heavily on which experts to deactivate, this paper tackles a neglected but critical question: How should we handle the weights of the surviving experts?
MoE models typically follow one of two weighting rules:
- Raw Probabilities: The expert branch weight is the raw softmax output (e.g., OLMoE, DeepSeek-V2). The sum of weights is usually $< 1$.
- Renormalized Weights: The top-k expert weights are rescaled to sum exactly to $1$ (e.g., Granite, Qwen3).
The authors investigate whether failing to respect these training-time choices during inference-time pruning is the hidden culprit behind poor pruning performance.
Methods
The authors developed a weight-sum model to analyze how different deactivation implementations change the "gain" of the expert branch relative to the residual connection. They compared four distinct implementations:
- Removal ($\alpha$): Masks disabled experts before the router's softmax, re-selects a new top-k, and recomputes weights using the original rule.
- Freezing ($\beta$): Keeps the original routing and weights but simply zeros out the contributions of disabled experts.
- Renormalization ($\gamma$): Keeps the original survivors but rescales their weights to sum to 1.
- Unnormalized Re-routing ($\delta$): Re-selects experts to fill vacated slots but weights them by their original raw softmax probabilities.
They tested these across five instruction-tuned model checkpoints (OLMoE-1B, DeepSeek-V2-Lite, Granite-3.1-1B/3B, and Qwen3-30B) to see how implementation choice interacts with the model's native training rule.
Datasets
- Calibration: The first 2,000 documents of the C4 shard were used to rank expert importance (via activation frequency).
- Evaluation:
- Language Modeling: WikiText-2 (perplexity).
- Reasoning & Knowledge: ARC (easy/challenge), OpenBookQA, HellaSwag, BoolQ, RTE, WinoGrande, and MMLU.
- Math: GSM8K (free-form generation).
Key Insights and Ideas
- The Balance of Branches: The expert branch and the residual branch exist in a delicate equilibrium. In "raw probability" models, the expert branch sum is often small. If you renormalize those survivors to sum to 1 ($\gamma$), you effectively "over-amplify" the expert branch, drowning out the residual signal.
- Implementation > Selection: On DeepSeek-V2-Lite, using the same set of disabled experts, "Freezing" ($\beta$) resulted in a manageable perplexity of 17.21, while "Renormalization" ($\gamma$) caused a total collapse to 258,621.64. The math of the weights mattered more than the experts themselves.
- Width Sensitivity: The penalty for a rule mismatch becomes even more severe as you reduce the number of active experts ($k$). As $k$ decreases, the total raw probability sum drops, making the "amplification" effect of renormalization even more extreme.
Key Results
- Catastrophic Mismatches: When the training rule was retained, perplexity stayed within 1.4x to 73x of the baseline. Under a rule mismatch, it skyrocketed to between 14x and 28,740x the baseline.
- Task Performance: At 50% deactivation, most models approached chance levels on complex tasks like MMLU. On GSM8K, strict-match accuracy fell to zero across the board, though qualitative failure modes differed: "Removal" led to echoing the prompt, while "Freezing" caused single-token loops.
- Predictive Power: The authors' weight-sum model successfully predicted which implementation would be most damaging for each model family (e.g., $\gamma$ for OLMoE/DeepSeek and $\delta$ for Granite/Qwen).
Limitations and Future Work
- Static Calibration: The study used a fixed C4 calibration set; results might vary with different data distributions.
- No Recovery Training: The paper focuses on "zero-shot" deactivation. It remains to be seen if short bursts of recovery fine-tuning can bridge the gap created by a weight-rule mismatch.
- Scale: While the authors tested up to 30B parameters, even larger models (like DeepSeek-V3 or Mixtral-Large) might exhibit different scaling behaviors regarding branch balance.
Summary updated