Aligning Training and Rollout Policies for End-to-End FP4 RL on Mixture-of-Experts LLMs

Hei Yi Mak, Hoang Le, Mehran Taghian Jazi, Yunke Peng, Huan Lin, Zhengzhi Tang, Zhexu Liu, Yao Wang, Anandharaju Durai Raju, Tanzila Rahman

TL;DR

This paper identifies why standard 4-bit quantization techniques for dense models fail when applied to Mixture-of-Experts (MoE) models during reinforcement learning (RL) post-training. The authors propose MoeResQ, a new recipe that enforces "symmetric recovery"—applying residual corrections to both the training and rollout policies—effectively closing the accuracy gap between 4-bit and full-precision (BF16) models.

What Problem Does This Paper Address?

Reinforcement Learning (RL) post-training (like the GRPO algorithm used in models like DeepSeek-R1) is the current gold standard for boosting the reasoning capabilities of Large Language Models (LLMs). However, it is incredibly resource-intensive. Quantizing these models to 4-bit (FP4) precision could drastically reduce memory and compute costs, but it often leads to significant accuracy drops.

While recent methods like RolloutResQ successfully recovered accuracy for dense models, the authors discovered these methods fail completely for Mixture-of-Experts (MoE) models. In some cases, applying these "dense-model" fixes actually made MoE performance worse than doing nothing at all. The paper investigates why MoE models are uniquely fragile under quantization and how to fix them.

Methods

The authors trace the failure to the router—the component in an MoE model that decides which "experts" (specialized sub-networks) to activate for a given token.

  1. The Mismatch Problem: In RL, we have a "rollout" policy (generating responses) and a "training" policy (updating weights). If quantization affects these two sides differently (asymmetric recovery), the routers might pick different experts for the same token. This creates a "rollout-training mismatch" that destabilizes learning.
  2. MoeResQ: To solve this, the authors propose a symmetric recovery recipe. They use Residual Correction (RC), which calculates the error introduced by quantization and adds it back via extra low-precision computations. Unlike previous methods that only corrected the rollout side, MoeResQ applies this correction to both the training and rollout sides.
  3. Precision Strategy: They use the HiF4 format for quantization. Crucially, they keep the routing layers in BF16 (since they are tiny—only ~0.04% of parameters—but highly sensitive) while quantizing the massive expert layers to FP4.

Datasets

The models were trained using the DAPO-Math-17K dataset with the GRPO algorithm. Evaluation was conducted across four challenging mathematical reasoning benchmarks:

  • AMC23
  • AIME24
  • AIME25
  • MATH500

Key Insights and Ideas

  • The Router as an Amplifier: In dense models, quantization error just slightly shifts the output values. In MoE models, a tiny error can flip the discrete decision of the router, sending the data to a completely different expert. This "flips" the policy, causing massive divergence.
  • Symmetry is Essential: The "aha moment" was realizing that the accuracy of the quantization matters less than the consistency between the training and inference sides.
  • Visualizing Alignment: The authors used the Pearson Correlation Coefficient (PCC) to measure how well the training and inference probabilities aligned. Asymmetric methods showed scattered, poor alignment, while MoeResQ restored a tight diagonal correlation similar to full precision.
Visualization of the alignment between training and inference probabilities across different quantization settings. Note how "RC Both" (MoeResQ) tightly mirrors the BF16 gold standard.

Key Results

MoeResQ successfully bridged the gap to full precision (BF16) where other methods failed:

  • Accuracy Recovery: On the AMC23 benchmark, naive FP4 (HiF4) dropped from 85.10% to 65.23%. The dense-model fix (RolloutResQ) actually dropped further to 62.34%. MoeResQ recovered this to 84.45%, nearly matching the BF16 original.
  • Expert-Only Quantization: When quantizing only the expert layers (which represent ~95% of the model), MoeResQ achieved results within 0.5% to 1.5% of full-precision accuracy across all benchmarks.
  • Alignment Metrics: MoeResQ restored the "top-k match fraction" (how often the same experts are chosen) from roughly 60-70% in asymmetric setups to over 85-98%, effectively eliminating the policy mismatch.

Limitations and Future Work

  • Simulated Quantization: Due to current hardware limitations for FP4, all experiments were conducted under simulated quantization. Real-world speedups will depend on future hardware support for these specific 4-bit formats.
  • Efficiency Bottlenecks: While MoeResQ is accurate, the authors note that future work should focus on making the symmetric residual correction even more computationally efficient to maximize the throughput gains of switching to 4-bit.
  • Scope: The study focused on mathematical reasoning; extending these findings to broader agentic workflows or general-purpose chat remains an open area.

Summary updated

Abstract

In this work, we study end-to-end FP4 reinforcement learning (RL) post-training for Mixture-of-Experts (MoE) large language models. Recent work on FP4 RL has introduced effective recovery recipes that restore accuracy under aggressive low-precision quantization for dense models, but we find that the same approach yields no improvement over the naive FP4 baseline in the MoE setting. We conduct an in-depth study to understand why. Our analysis identifies the root cause in the routing mechanism unique to MoE: because the router makes discrete expert-selection decisions, training-inference policy alignment becomes acutely sensitive to whether the same recovery technique is applied symmetrically on the training and inference sides. When an asymmetric recipe is used-including the dense-model approach, which corrects only the rollout-training-inference alignment deteriorates sharply, as measured by the Pearson correlation coefficient between training and inference probabilities, and router selection matching rates. Building on this finding, we propose MoeResQ, a quantization-recovery recipe for MoE RL post-training that enforces symmetry between the training- and inference-side corrections. MoeResQ restores training-inference alignment and substantially closes the accuracy gap to full precision, establishing symmetric recovery as the key principle in FP4 RL for MoE models.
BibTeX
@misc{mak2026aligning,
  author = {Hei Yi Mak and Hoang Le and Mehran Taghian Jazi and Yunke Peng and Huan Lin and Zhengzhi Tang and Zhexu Liu and Yao Wang and Yaoyuan Wang and Anandharaju Durai Raju and Tanzila Rahman},
  title = {{Aligning Training and Rollout Policies for End-to-End FP4 RL on Mixture-of-Experts LLMs}},
  year = {2026},
  eprint = {2611.04376},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04376},
}
Figure