TL;DR
While the popular matrix-aware optimizer Muon excels at LLM pretraining via uniform spectral whitening, this paper reveals that it fails dramatically in post-pretraining regimes like Vision-Language-Action (VLA) models and Reinforcement Learning with Verifiable Rewards (RLVR). To fix this, the authors introduce Pion, a drop-in replacement that uses a novel "Promotion+Suppression" polynomial mechanism to act as a spectral high-pass filter, suppressing gradient noise while retaining critical directions without adding any computational overhead.
What Problem Does This Paper Address?
Muon has recently made waves in LLM pretraining by departing from traditional element-wise optimization (like AdamW) in favor of a matrix-aware approach. It leverages Newton-Schulz (NS) iterations to drive all singular values of the momentum matrix to 1—a process known as uniform spectral whitening.
While this uniform whitening encourages broad exploration during pretraining, the authors identify a critical "spectral mismatch" when applying Muon to other increasingly important training paradigms:
- Cross-Modality VLA Training: Action-module gradients are inherently low-rank compared to high-rank vision and language features.
- RLVR Post-Training: Policy gradients suffer from a very low signal-to-noise ratio (SNR).
In both cases, Muon's uniform whitening inflates the noisy "tail" of the singular value spectrum to the same magnitude as the informative "head." This amplifies gradient noise, leading to suboptimal performance in VLA models and catastrophic model collapse in RLVR.
Methods
The authors propose Pion (sPectral hIgh-pass Optimization on momeNtum) as a computationally efficient, drop-in alternative to Muon. Instead of uniformly whitening the gradient spectrum, Pion splits the Newton-Schulz iterations into a two-stage polynomial mapping:
- Promotion Stage: Monotonically amplifies singular values to lift dominant components.
- Suppression Stage: Acts as a soft high-pass filter, anchoring leading singular values near 1 while driving noisy tail components toward 0.
Furthermore, to respect the heterogeneous update scales required across attention heads inherited from pretraining, Pion introduces a per-head mode. This mode reshapes attention projections along the head dimension and applies the high-pass filter independently per head at zero extra computational cost.
Datasets
The authors evaluated Pion across two distinct domains:
- VLA Training: Evaluated in simulation using the four LIBERO suites (Object, Spatial, Goal, Long) and LIBERO-Plus (which introduces environmental and robotic perturbations). Real-world physical deployment was tested on a Franka Research 3 robot under the DROID setup.
- RLVR Post-Training: Evaluated using Qwen3-1.7B and Qwen3-4B models on the MATH (levels 3–5, tested on MATH500) and GSM8K datasets using GRPO and GMPO reinforcement learning algorithms.
Key Insights and Ideas
- Structural Rank Heterogeneity: In VLA models, the vision and language modules process high-entropy data, whereas the action head outputs low-dimensional control vectors (e.g., 7D end-effector movements). Uniformly whitening the action head over-amplifies non-generalizable noise directions.
- RLVR is Inherently Noisier than SFT: Policy gradients in RLVR have significantly lower SNR than Supervised Fine-Tuning (SFT) due to sparse trajectory-level rewards and stabilization mechanisms like clipping or importance sampling.
- Preserving Pretrained Heterogeneity: Attention layers exhibit highly specialized per-head weight norms after pretraining. Treating a layer as a single block (as Muon does) erases this structure, whereas Pion's per-head mode preserves it.
Key Results
- VLA Success Rates: On the LIBERO Object benchmark, Pion achieved a 100% success rate after just 1,500 training steps using a VLA-Adapter, compared to 97.0% for Muon and a dismal 32.2% for AdamW. On the real Franka robot, Pion achieved an average success rate of 85.6% across three grasp-and-place tasks, vastly outperforming Muon (38.9%) and AdamW (31.1%).
- RLVR Stability: While Muon completely collapsed to 0% accuracy on MATH and GSM8K benchmarks due to noise amplification, Pion successfully stabilized training and consistently outperformed AdamW across both GRPO and GMPO algorithms.
Limitations and Future Work
Pion is explicitly tailored for regimes where informative gradients concentrate in a few leading singular values. Because it filters out the tail, it is expected to underperform Muon in standard LLM pretraining, where uniform whitening and maximum spectral exploration are highly beneficial.
The authors note that future work should explore how to dynamically adapt the high-pass cutoff to seamlessly recover Muon's exploratory behavior during pretraining while retaining Pion's robust noise resilience during post-training and cross-modality adaptation.
Summary updated