DSPO: Stable and Efficient Policy Optimization for Agentic Search and Reasoning

Chenyang Gu, Yewen Pu, Bruce Yang, Xiaofan Li, Huan Gao

Introduction

Large Language Models (LLMs) are increasingly expected to act as autonomous agents, querying external search engines to overcome their inherent knowledge cutoffs. However, training these agents via Reinforcement Learning (RL) is notoriously difficult. Standard RL methods often suffer from an objective mismatch, where token-level optimization conflicts with sequence-level rewards, and sparse rewards, where a lack of variance in training batches prevents effective learning. These issues frequently lead to high-variance gradients and "catastrophic policy collapse."

The DSPO Framework

To address these instabilities, we introduce Dynamic-filter Sequence-level Policy Optimization (DSPO). DSPO aligns the optimization objective with the actual task outcome and ensures a robust learning signal through two primary innovations:

  1. Sequence-level Policy Optimization: Instead of noisy token-level importance ratios used in standard PPO or GRPO, DSPO employs a sequence-level importance ratio. This is calculated as the geometric mean of token-level ratios across the entire trajectory, ensuring the unit of optimization matches the unit of reward (the final answer).
  2. Dynamic Outcome-based Filtering: To combat the sparse reward problem, DSPO implements a filtering mechanism that retains only groups of trajectories containing a mix of successful and unsuccessful outcomes. By ensuring every training batch has internal variance, the algorithm guarantees a meaningful "advantage signal," transforming a sparse environment into a dense, high-quality learning signal.
The DSPO framework integrates a search engine with sequence-level policy optimization and dynamic outcome-based filtering to stabilize agent training.

Crucially, the model is trained purely through RL to interleave search and reasoning, removing the need for expensive, human-annotated supervised demonstration data.

Experimental Evaluation

The model was trained on a composite dataset consisting of the training splits of Natural Questions (NQ) and HotpotQA. Evaluation was conducted across seven diverse benchmarks to test generalization:

  • Single-hop/General Knowledge: NQ, TriviaQA, and PopQA.
  • Complex Multi-hop Reasoning: HotpotQA, 2WikiMultiHopQA, and Musique.
  • Search-Resistant Tasks: Bamboogle.

Key Results and Stability

DSPO demonstrates remarkable stability compared to existing baselines. While token-level methods like vanilla GRPO often experience sudden performance drops, DSPO exhibits a smooth, monotonic increase in reward throughout the entire training process.

Training reward curves demonstrate that DSPO achieves superior stability and higher final rewards compared to GRPO and various ablation models, avoiding the policy collapse seen in token-level methods.

Quantitatively, the DSPO-trained 7B model achieved a 34.1% relative improvement over the Search-R1 (GRPO 7B) baseline. A striking insight from the results is that algorithmic robustness can be more impactful than raw parameter count: the DSPO 7B model outperformed a much larger 14B model on the complex HotpotQA benchmark by nearly 9% relative (0.613 vs 0.563).

Discussion and Emergent Behaviors

Through pure RL, the model developed sophisticated agentic strategies without explicit programming. These emergent behaviors include recognizing irrelevant search results, reformulating failed queries, and performing multi-turn verification of facts. The success of DSPO suggests that the current bottleneck for agentic LLMs is not necessarily model size, but rather the stability of the training objective.

Conclusion

DSPO provides a stable and efficient path for training LLMs as search agents. While highly effective, future work will explore integrating advanced dense retrievers and extending the framework to multi-tool environments, such as autonomous code generation and complex mathematics.

Summary updated

Abstract

Enhancing LLMs with the ability to actively search external knowledge is crucial for complex and real-world tasks. Current approaches either rely on prompting to elicit the model's innate agent capabilities, or suffer from performance ceilings and collapse when applying RL to complex interactive tasks, leaving their true agentic potential untapped. To address this, we introduce $\textbf{D}$ynamic-filter $\textbf{S}$equence-level $\textbf{P}$olicy $\textbf{O}$ptimization (DSPO), an improved RL algorithm designed for robust agent training through sequence-level optimization and dynamic sample filtering. We train our model purely through RL to interleave multi-turn search and reasoning, obviating the need for supervised demonstration data. Across multiple QA benchmarks, our DSPO-trained 7B model improves over a comparable previous work by $\textbf{34.1\%}$, and even outperforms the 14B model from previous work in complex multihop QA such as HotpotQA by nearly $\textbf{9\% relative}$, maintaining exceptional training stability.
BibTeX
@misc{gu2025dspo,
  author = {Chenyang Gu and Yewen Pu and Bruce Yang and Xiaofan Li and Huan Gao},
  title = {{DSPO: Stable and Efficient Policy Optimization for Agentic Search and Reasoning}},
  year = {2025},
  eprint = {2510.09255},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.09255},
}
Figure