Introduction
Reinforcement learning (RL) has become a cornerstone for enhancing the reasoning capabilities of Large Language Models (LLMs), as seen in recent breakthroughs like DeepSeek-R1 and Qwen-2.5-Math. Most state-of-the-art approaches utilize group-based policy optimization (e.g., GRPO or GSPO), where models learn by comparing multiple generated responses. However, these methods are often plagued by training instability caused by high importance ratio variance. When the current policy deviates significantly from the behavior policy, updates can become excessively noisy, leading to "training collapse."
To mitigate this, existing frameworks rely on hard clipping, which zeroes out gradients if the importance ratio falls outside a predefined range (typically $[0.8, 1.2]$). This "all-or-nothing" approach is inherently brittle. In sequence-level optimization (GSPO), a single outlier token can invalidate an entire sequence's learning signal. In token-level optimization (GRPO), hard clipping ignores potentially valuable information just outside the boundary.
The SAPO Framework
The Qwen team at Alibaba proposes Soft Adaptive Policy Optimization (SAPO) to address these limitations. SAPO replaces the sharp cutoff of hard clipping with a Soft Adaptive Gate, a sigmoid-shaped function that weights updates based on the importance ratio $r$.
Continuous Trust Region
Unlike hard clipping, the soft gate provides a continuous transition. Near the on-policy point ($r=1$), the gate remains fully open. As the ratio deviates, the gate closes smoothly, allowing the model to learn from moderately off-policy data while still suppressing extreme outliers. This creates a more robust "continuous trust region" for policy updates.
Asymmetric Temperature Control
A key innovation in SAPO is the use of asymmetric temperatures ($\tau$) for positive and negative updates. The authors found that negative updates (telling the model what not to say) are a significant source of noise in LLMs with large vocabularies. By setting a higher temperature for negative tokens ($\tau_{neg} > \tau_{pos}$), SAPO causes their gradients to decay faster, effectively dampening the noise that could otherwise destabilize the training process.
Stability and Empirical Results
SAPO's design is particularly beneficial for complex architectures like Mixture-of-Experts (MoE), which are notoriously difficult to stabilize during RL due to routing heterogeneity. By selectively down-weighting only the problematic tokens rather than discarding entire sequences, SAPO maintains sequence coherence while allowing for token-level adaptivity.
Key Performance Metrics
The effectiveness of SAPO was validated across several demanding benchmarks:
- Mathematical Reasoning: SAPO consistently reached higher peak performance than both GSPO and GRPO on AIME25 and HMMT25.
- Coding and Logic: Significant gains were observed on LiveCodeBench v6 and ZebraLogic.
- Multimodal Tasks: The method was instrumental in training the Qwen3-VL series, showing consistent improvements in vision-language reasoning (e.g., MathVision).
Conclusion
Soft Adaptive Policy Optimization represents a significant step forward in making RL fine-tuning for LLMs more reliable. By transitioning from brittle hard clipping to a smooth, asymmetric soft gate, SAPO achieves superior stability and performance across a wide range of tasks and model architectures. This approach eliminates the need for complex engineering workarounds, such as routing replay for MoE models, providing a more principled path toward advanced AI reasoning.
Summary updated