TL;DR
Quantizing Mixture-of-Experts (MoE) models often breaks their routing logic, as tiny numerical errors can cause tokens to be sent to the wrong experts. This paper introduces VSRAQ, a post-training quantization objective that aligns both the numerical values and the structural relationships (ordering and margins) of router outputs. By maintaining routing consistency before and after quantization, VSRAQ significantly reduces performance degradation on complex reasoning tasks without adding any inference-time overhead.
What Problem Does This Paper Address?
Mixture-of-Experts (MoE) architectures scale the capacity of large language models efficiently by activating only a small subset (top-$k$) of specialized parameters (experts) for each token. However, deploying these models remains a massive challenge because the entire set of expert parameters must still reside in memory. Post-Training Quantization (PTQ) is the go-to solution to compress these models, but standard dense quantization objectives fall short here.
Traditional PTQ focuses on minimizing numerical reconstruction errors within a fixed computation graph. In contrast, MoE layers rely on input-dependent routing. Even tiny quantization-induced perturbations in hidden states or router logits can cause routing instability—flipping the top-$k$ expert selection. This alters the actual computation path, causing errors to compound rapidly over long generations and leading to severe performance degradation.
Methods
The authors propose Value-and-Structure Routing Alignment for Quantization (VSRAQ). Rather than acting as a standalone quantization algorithm, VSRAQ is a calibration objective designed to augment existing PTQ frameworks (like AutoRound). It introduces a specialized router alignment loss consisting of two complementary components:
- Value Alignment: This matches routing-relevant scores before and after quantization. Beyond standard top-$k$ logit matching, it introduces a novel sigmoid-aware score preservation loss. Because modern MoEs (e.g., DeepSeek-V3) utilize sigmoid-based routing, scores near $0.5$ are highly sensitive to small logit changes due to the steepness of the sigmoid curve. VSRAQ applies a sensitivity weight to focus optimization on these volatile, routing-critical regions.
- Structure Alignment: This preserves the structural relationships of the router outputs. It enforces top-$k$ order preservation (penalizing rank inversions among selected experts) and boundary preservation (maintaining the margin between the $k$-th selected expert and the top non-selected experts to ensure the boundary remains stable under noise).
Datasets
- Calibration Set: The authors utilize a compact, reasoning-oriented dataset called the NVIDIA Reasoning calibration set, sampled from OpenCodeReasoning, OpenScienceReasoning-2, and OpenMathReasoning.
- Evaluation Benchmarks: Models are evaluated across three core capabilities using
lm-evaluation-harness: - Perplexity: WikiText-2.
- Generation-based answer extraction: MMLU-Pro, GPQA-Diamond, and GSM-8K (evaluated using greedy decoding to eliminate sampling randomness).
- Likelihood-based multiple-choice: ARC-C, ARC-E, BoolQ, HellaSwag, MMLU, PIQA, TruthfulQA, and WinoGrande.
Key Insights and Ideas
- Routing is Structural, Not Just Numerical: Simply matching logit values isn't enough. The relative ordering of experts and the size of the selection margin are what truly dictate routing stability. If a quantization error narrows the margin between the $k$-th and $(k+1)$-th expert, the model becomes highly vulnerable to rerouting.
- The S-Curve Vulnerability: In sigmoid routing, errors near $0.5$ are catastrophic compared to errors near saturated regions ($0$ or $1$). VSRAQ's sensitivity weighting prevents the optimization process from wasting capacity on saturated scores, focusing instead on the unstable decision thresholds.
- The Butterfly Effect in Generation: On simple multiple-choice tasks, routing mismatches matter less. However, on free-form generation tasks, a single routing error alters the subsequent hidden states, which corrupts future routing choices. This compounding drift is why conventional quantization fails so noticeably on complex reasoning tasks.
Key Results
The authors validated VSRAQ on two major foundation models: Solar-Open-100B (a large Transformer MoE) and Nemotron-3-Nano-30B-A3B (a hybrid Mamba-Transformer MoE), evaluating both weight-only (W4A16) and weight-activation (NVFP4) 4-bit quantization.
- Superior Performance: VSRAQ consistently outperformed reconstruction-only (AutoRound) and value-only (TopK-MSE) baselines. The gains were most pronounced in generation-based extraction tasks where errors compound.
- Expert-Selection Consistency: By directly measuring the Jaccard similarity of selected experts between the full-precision and quantized models, VSRAQ demonstrated superior layer-wise routing consistency, achieving up to an $11.29\%$ relative improvement over the baseline.
Limitations and Future Work
- Focus on 4-bit Quantization: The paper primarily explores W4A16 and NVFP4 regimes, where quantization noise is relatively mild.
- Ultra-low Bit-widths: The authors note that routing instability is bound to be far more severe in aggressive 2-bit or 3-bit quantization regimes. Extending and testing VSRAQ in these ultra-low bit-width settings is left for future work.
- Architectural Scope: While the method proved effective on both standard Transformer and hybrid Mamba-Transformer MoEs, broader evaluations across an even wider variety of MoE routing mechanisms remain an open direction.
Summary updated