Soft Adaptive Policy Optimization

Chang Gao, Chujie Zheng, Xionghui Chen, Kai Dang, Shixuan Liu, Bowen Yu, An Yang, Shuai Bai, Jingren Zhou, Junyang LinarXiv 2025 · DOI · arXiv

Introduction

Reinforcement learning (RL) has become a cornerstone for enhancing the reasoning capabilities of Large Language Models (LLMs), as seen in recent breakthroughs like DeepSeek-R1 and Qwen-2.5-Math. Most state-of-the-art approaches utilize group-based policy optimization (e.g., GRPO or GSPO), where models learn by comparing multiple generated responses. However, these methods are often plagued by training instability caused by high importance ratio variance. When the current policy deviates significantly from the behavior policy, updates can become excessively noisy, leading to "training collapse."

To mitigate this, existing frameworks rely on hard clipping, which zeroes out gradients if the importance ratio falls outside a predefined range (typically $[0.8, 1.2]$). This "all-or-nothing" approach is inherently brittle. In sequence-level optimization (GSPO), a single outlier token can invalidate an entire sequence's learning signal. In token-level optimization (GRPO), hard clipping ignores potentially valuable information just outside the boundary.

The SAPO Framework

The Qwen team at Alibaba proposes Soft Adaptive Policy Optimization (SAPO) to address these limitations. SAPO replaces the sharp cutoff of hard clipping with a Soft Adaptive Gate, a sigmoid-shaped function that weights updates based on the importance ratio $r$.

Continuous Trust Region

Unlike hard clipping, the soft gate provides a continuous transition. Near the on-policy point ($r=1$), the gate remains fully open. As the ratio deviates, the gate closes smoothly, allowing the model to learn from moderately off-policy data while still suppressing extreme outliers. This creates a more robust "continuous trust region" for policy updates.

Comparison of the Soft Adaptive Gate weighting function and the resulting objective value against hard clipping.

Asymmetric Temperature Control

A key innovation in SAPO is the use of asymmetric temperatures ($\tau$) for positive and negative updates. The authors found that negative updates (telling the model what not to say) are a significant source of noise in LLMs with large vocabularies. By setting a higher temperature for negative tokens ($\tau_{neg} > \tau_{pos}$), SAPO causes their gradients to decay faster, effectively dampening the noise that could otherwise destabilize the training process.

Stability and Empirical Results

SAPO's design is particularly beneficial for complex architectures like Mixture-of-Experts (MoE), which are notoriously difficult to stabilize during RL due to routing heterogeneity. By selectively down-weighting only the problematic tokens rather than discarding entire sequences, SAPO maintains sequence coherence while allowing for token-level adaptivity.

Empirical distribution of importance ratios and variance bounds, demonstrating the stability of the SAPO optimization process.

Key Performance Metrics

The effectiveness of SAPO was validated across several demanding benchmarks:

  • Mathematical Reasoning: SAPO consistently reached higher peak performance than both GSPO and GRPO on AIME25 and HMMT25.
  • Coding and Logic: Significant gains were observed on LiveCodeBench v6 and ZebraLogic.
  • Multimodal Tasks: The method was instrumental in training the Qwen3-VL series, showing consistent improvements in vision-language reasoning (e.g., MathVision).

Conclusion

Soft Adaptive Policy Optimization represents a significant step forward in making RL fine-tuning for LLMs more reliable. By transitioning from brittle hard clipping to a smooth, asymmetric soft gate, SAPO achieves superior stability and performance across a wide range of tasks and model architectures. This approach eliminates the need for complex engineering workarounds, such as routing replay for MoE models, providing a more principled path toward advanced AI reasoning.

Summary updated

Abstract

Reinforcement learning (RL) plays an increasingly important role in enhancing the reasoning capabilities of large language models (LLMs), yet stable and performant policy optimization remains challenging. Token-level importance ratios often exhibit high variance-a phenomenon exacerbated in Mixture-of-Experts models-leading to unstable updates. Existing group-based policy optimization methods, such as GSPO and GRPO, alleviate this problem via hard clipping, making it difficult to maintain both stability and effective learning. We propose Soft Adaptive Policy Optimization (SAPO), which replaces hard clipping with a smooth, temperature-controlled gate that adaptively attenuates off-policy updates while preserving useful learning signals. Compared with GSPO and GRPO, SAPO is both sequence-coherent and token-adaptive. Like GSPO, SAPO maintains sequence-level coherence, but its soft gating forms a continuous trust region that avoids the brittle hard clipping band used in GSPO. When a sequence contains a few highly off-policy tokens, GSPO suppresses all gradients for that sequence, whereas SAPO selectively down-weights only the offending tokens and preserves the learning signal from the near-on-policy ones, improving sample efficiency. Relative to GRPO, SAPO replaces hard token-level clipping with smooth, temperature-controlled scaling, enabling more informative and stable updates. Empirical results on mathematical reasoning benchmarks indicate that SAPO exhibits improved training stability and higher Pass@1 performance under comparable training budgets. Moreover, we employ SAPO to train the Qwen3-VL model series, demonstrating that SAPO yields consistent performance gains across diverse tasks and different model sizes. Overall, SAPO provides a more reliable, scalable, and effective optimization strategy for RL training of LLMs.
BibTeX
@misc{gao2025soft,
  author = {Chang Gao and Chujie Zheng and Xionghui Chen and Kai Dang and Shixuan Liu and Bowen Yu and An Yang and Shuai Bai and Jingren Zhou and Junyang Lin},
  title = {{Soft Adaptive Policy Optimization}},
  year = {2025},
  eprint = {2511.20347},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.20347},
}
Figure