Introduction
Large Language Models (LLMs) are increasingly expected to act as autonomous agents, querying external search engines to overcome their inherent knowledge cutoffs. However, training these agents via Reinforcement Learning (RL) is notoriously difficult. Standard RL methods often suffer from an objective mismatch, where token-level optimization conflicts with sequence-level rewards, and sparse rewards, where a lack of variance in training batches prevents effective learning. These issues frequently lead to high-variance gradients and "catastrophic policy collapse."
The DSPO Framework
To address these instabilities, we introduce Dynamic-filter Sequence-level Policy Optimization (DSPO). DSPO aligns the optimization objective with the actual task outcome and ensures a robust learning signal through two primary innovations:
- Sequence-level Policy Optimization: Instead of noisy token-level importance ratios used in standard PPO or GRPO, DSPO employs a sequence-level importance ratio. This is calculated as the geometric mean of token-level ratios across the entire trajectory, ensuring the unit of optimization matches the unit of reward (the final answer).
- Dynamic Outcome-based Filtering: To combat the sparse reward problem, DSPO implements a filtering mechanism that retains only groups of trajectories containing a mix of successful and unsuccessful outcomes. By ensuring every training batch has internal variance, the algorithm guarantees a meaningful "advantage signal," transforming a sparse environment into a dense, high-quality learning signal.
Crucially, the model is trained purely through RL to interleave search and reasoning, removing the need for expensive, human-annotated supervised demonstration data.
Experimental Evaluation
The model was trained on a composite dataset consisting of the training splits of Natural Questions (NQ) and HotpotQA. Evaluation was conducted across seven diverse benchmarks to test generalization:
- Single-hop/General Knowledge: NQ, TriviaQA, and PopQA.
- Complex Multi-hop Reasoning: HotpotQA, 2WikiMultiHopQA, and Musique.
- Search-Resistant Tasks: Bamboogle.
Key Results and Stability
DSPO demonstrates remarkable stability compared to existing baselines. While token-level methods like vanilla GRPO often experience sudden performance drops, DSPO exhibits a smooth, monotonic increase in reward throughout the entire training process.
Quantitatively, the DSPO-trained 7B model achieved a 34.1% relative improvement over the Search-R1 (GRPO 7B) baseline. A striking insight from the results is that algorithmic robustness can be more impactful than raw parameter count: the DSPO 7B model outperformed a much larger 14B model on the complex HotpotQA benchmark by nearly 9% relative (0.613 vs 0.563).
Discussion and Emergent Behaviors
Through pure RL, the model developed sophisticated agentic strategies without explicit programming. These emergent behaviors include recognizing irrelevant search results, reformulating failed queries, and performing multi-turn verification of facts. The success of DSPO suggests that the current bottleneck for agentic LLMs is not necessarily model size, but rather the stability of the training objective.
Conclusion
DSPO provides a stable and efficient path for training LLMs as search agents. While highly effective, future work will explore integrating advanced dense retrievers and extending the framework to multi-tool environments, such as autonomous code generation and complex mathematics.
Summary updated