TL;DR
This paper identifies why standard 4-bit quantization techniques for dense models fail when applied to Mixture-of-Experts (MoE) models during reinforcement learning (RL) post-training. The authors propose MoeResQ, a new recipe that enforces "symmetric recovery"—applying residual corrections to both the training and rollout policies—effectively closing the accuracy gap between 4-bit and full-precision (BF16) models.
What Problem Does This Paper Address?
Reinforcement Learning (RL) post-training (like the GRPO algorithm used in models like DeepSeek-R1) is the current gold standard for boosting the reasoning capabilities of Large Language Models (LLMs). However, it is incredibly resource-intensive. Quantizing these models to 4-bit (FP4) precision could drastically reduce memory and compute costs, but it often leads to significant accuracy drops.
While recent methods like RolloutResQ successfully recovered accuracy for dense models, the authors discovered these methods fail completely for Mixture-of-Experts (MoE) models. In some cases, applying these "dense-model" fixes actually made MoE performance worse than doing nothing at all. The paper investigates why MoE models are uniquely fragile under quantization and how to fix them.
Methods
The authors trace the failure to the router—the component in an MoE model that decides which "experts" (specialized sub-networks) to activate for a given token.
- The Mismatch Problem: In RL, we have a "rollout" policy (generating responses) and a "training" policy (updating weights). If quantization affects these two sides differently (asymmetric recovery), the routers might pick different experts for the same token. This creates a "rollout-training mismatch" that destabilizes learning.
- MoeResQ: To solve this, the authors propose a symmetric recovery recipe. They use Residual Correction (RC), which calculates the error introduced by quantization and adds it back via extra low-precision computations. Unlike previous methods that only corrected the rollout side, MoeResQ applies this correction to both the training and rollout sides.
- Precision Strategy: They use the HiF4 format for quantization. Crucially, they keep the routing layers in BF16 (since they are tiny—only ~0.04% of parameters—but highly sensitive) while quantizing the massive expert layers to FP4.
Datasets
The models were trained using the DAPO-Math-17K dataset with the GRPO algorithm. Evaluation was conducted across four challenging mathematical reasoning benchmarks:
- AMC23
- AIME24
- AIME25
- MATH500
Key Insights and Ideas
- The Router as an Amplifier: In dense models, quantization error just slightly shifts the output values. In MoE models, a tiny error can flip the discrete decision of the router, sending the data to a completely different expert. This "flips" the policy, causing massive divergence.
- Symmetry is Essential: The "aha moment" was realizing that the accuracy of the quantization matters less than the consistency between the training and inference sides.
- Visualizing Alignment: The authors used the Pearson Correlation Coefficient (PCC) to measure how well the training and inference probabilities aligned. Asymmetric methods showed scattered, poor alignment, while MoeResQ restored a tight diagonal correlation similar to full precision.
Key Results
MoeResQ successfully bridged the gap to full precision (BF16) where other methods failed:
- Accuracy Recovery: On the AMC23 benchmark, naive FP4 (HiF4) dropped from 85.10% to 65.23%. The dense-model fix (RolloutResQ) actually dropped further to 62.34%. MoeResQ recovered this to 84.45%, nearly matching the BF16 original.
- Expert-Only Quantization: When quantizing only the expert layers (which represent ~95% of the model), MoeResQ achieved results within 0.5% to 1.5% of full-precision accuracy across all benchmarks.
- Alignment Metrics: MoeResQ restored the "top-k match fraction" (how often the same experts are chosen) from roughly 60-70% in asymmetric setups to over 85-98%, effectively eliminating the policy mismatch.
Limitations and Future Work
- Simulated Quantization: Due to current hardware limitations for FP4, all experiments were conducted under simulated quantization. Real-world speedups will depend on future hardware support for these specific 4-bit formats.
- Efficiency Bottlenecks: While MoeResQ is accurate, the authors note that future work should focus on making the symmetric residual correction even more computationally efficient to maximize the throughput gains of switching to 4-bit.
- Scope: The study focused on mathematical reasoning; extending these findings to broader agentic workflows or general-purpose chat remains an open area.
Summary updated