Matching the Training-Time Weight Rule in Inference-Time Expert Deactivation

Hao Cheng, Zhiyu Yang, Bo Ma, Hanzhang JiaarXiv 2026 · arXiv

TL;DR

When deactivating experts in Mixture-of-Experts (MoE) models to save on deployment costs, the mathematical rule used to weight the remaining experts must strictly match the rule used during the model's original training. Mismatching these rules—such as renormalizing weights in a model trained on raw probabilities—can lead to catastrophic performance degradation, increasing perplexity by as much as 28,000x.

What Problem Does This Paper Address?

Mixture-of-Experts (MoE) architectures like DeepSeek-V2 and OLMoE are highly efficient because they only activate a fraction of their parameters for any given token. However, the full "expert bank" still occupies massive amounts of VRAM. To reduce these deployment costs, researchers often use expert deactivation (pruning or offloading).

While existing literature focuses heavily on which experts to deactivate, this paper tackles a neglected but critical question: How should we handle the weights of the surviving experts?

MoE models typically follow one of two weighting rules:

  1. Raw Probabilities: The expert branch weight is the raw softmax output (e.g., OLMoE, DeepSeek-V2). The sum of weights is usually $< 1$.
  2. Renormalized Weights: The top-k expert weights are rescaled to sum exactly to $1$ (e.g., Granite, Qwen3).

The authors investigate whether failing to respect these training-time choices during inference-time pruning is the hidden culprit behind poor pruning performance.

Methods

The authors developed a weight-sum model to analyze how different deactivation implementations change the "gain" of the expert branch relative to the residual connection. They compared four distinct implementations:

  • Removal ($\alpha$): Masks disabled experts before the router's softmax, re-selects a new top-k, and recomputes weights using the original rule.
  • Freezing ($\beta$): Keeps the original routing and weights but simply zeros out the contributions of disabled experts.
  • Renormalization ($\gamma$): Keeps the original survivors but rescales their weights to sum to 1.
  • Unnormalized Re-routing ($\delta$): Re-selects experts to fill vacated slots but weights them by their original raw softmax probabilities.

They tested these across five instruction-tuned model checkpoints (OLMoE-1B, DeepSeek-V2-Lite, Granite-3.1-1B/3B, and Qwen3-30B) to see how implementation choice interacts with the model's native training rule.

Figure 1: Four implementations in a ten-expert example. The original top-8 set occupies slots 0-7, with slots 6 and 7 deactivated (red crosses). Re-routing fills the vacated slots with replacement experts, freezing retains the surviving weights, and renormalization rescales them. Unnormalized re-routing uses the original router probabilities.

Datasets

  • Calibration: The first 2,000 documents of the C4 shard were used to rank expert importance (via activation frequency).
  • Evaluation:
    • Language Modeling: WikiText-2 (perplexity).
  • Reasoning & Knowledge: ARC (easy/challenge), OpenBookQA, HellaSwag, BoolQ, RTE, WinoGrande, and MMLU.
  • Math: GSM8K (free-form generation).

Key Insights and Ideas

  • The Balance of Branches: The expert branch and the residual branch exist in a delicate equilibrium. In "raw probability" models, the expert branch sum is often small. If you renormalize those survivors to sum to 1 ($\gamma$), you effectively "over-amplify" the expert branch, drowning out the residual signal.
  • Implementation > Selection: On DeepSeek-V2-Lite, using the same set of disabled experts, "Freezing" ($\beta$) resulted in a manageable perplexity of 17.21, while "Renormalization" ($\gamma$) caused a total collapse to 258,621.64. The math of the weights mattered more than the experts themselves.
  • Width Sensitivity: The penalty for a rule mismatch becomes even more severe as you reduce the number of active experts ($k$). As $k$ decreases, the total raw probability sum drops, making the "amplification" effect of renormalization even more extreme.
Perplexity at 50% deactivation shows that the highest loss always occurs under a rule mismatch (e.g., renormalization for raw-probability models).

Key Results

  • Catastrophic Mismatches: When the training rule was retained, perplexity stayed within 1.4x to 73x of the baseline. Under a rule mismatch, it skyrocketed to between 14x and 28,740x the baseline.
  • Task Performance: At 50% deactivation, most models approached chance levels on complex tasks like MMLU. On GSM8K, strict-match accuracy fell to zero across the board, though qualitative failure modes differed: "Removal" led to echoing the prompt, while "Freezing" caused single-token loops.
  • Predictive Power: The authors' weight-sum model successfully predicted which implementation would be most damaging for each model family (e.g., $\gamma$ for OLMoE/DeepSeek and $\delta$ for Granite/Qwen).
OLMoE performance across different active expert counts ($k$). Renormalization causes significantly worse degradation as $k$ decreases.

Limitations and Future Work

  • Static Calibration: The study used a fixed C4 calibration set; results might vary with different data distributions.
  • No Recovery Training: The paper focuses on "zero-shot" deactivation. It remains to be seen if short bursts of recovery fine-tuning can bridge the gap created by a weight-rule mismatch.
  • Scale: While the authors tested up to 30B parameters, even larger models (like DeepSeek-V3 or Mixtral-Large) might exhibit different scaling behaviors regarding branch balance.

Summary updated

Abstract

Expert deactivation at inference time changes both token routing and expert weights, and its effect on performance depends on how these changes are implemented. We compare removal, freezing, remaining-expert renormalization, and unnormalized re-routing on identical disabled sets across five instruction-tuned checkpoints from four model families, with a base OLMoE checkpoint as an additional comparison. We develop a weight-sum model that relates the loss of expert contributions and changes in branch scale to the weight rule used during training. With half of the experts deactivated according to activation-based rankings, remaining-expert renormalization yields the highest perplexity in models trained with raw router probabilities, whereas unnormalized re-routing yields the highest perplexity in models trained with normalized weights. Perplexity ranges from 1.4 to 73 times the intact baseline when the training rule is retained, and from 14 to 28,740 times the baseline under a rule mismatch. Task outcomes vary across evaluation types: on OLMoE, GSM8K strict-match accuracy falls to zero under all four implementations. Extending the model to changes in the number of active experts yields a further prediction, supported by the OLMoE experiments: the perplexity penalty from additional renormalization increases as fewer experts are activated.
BibTeX
@misc{cheng2026matching,
  author = {Hao Cheng and Zhiyu Yang and Bo Ma and Hanzhang Jia},
  title = {{Matching the Training-Time Weight Rule in Inference-Time Expert Deactivation}},
  year = {2026},
  eprint = {2611.05804},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.05804},
}
Figure