Large Language Models (LLMs) have become remarkably good at "thinking" through complex math and logic problems. However, in the real world, reasoning isn't just internal—it involves interacting with tools like search engines, web browsers, and code interpreters. Training models to use these tools effectively via Reinforcement Learning (RL) has traditionally been a "coarse" process, treating an entire multi-step interaction as a single unit.
A new paper from researchers at Renmin University and Kuaishou Technology introduces Agentic Reinforced Policy Optimization (ARPO). ARPO shifts the focus from whole-trajectory exploration to fine-grained, step-level exploration, allowing models to learn how to use tools more efficiently and accurately.
TL;DR
ARPO is a novel RL algorithm that improves LLM tool-use by adaptively branching exploration at moments of high uncertainty (token entropy) following tool calls. It achieves superior performance across 13 benchmarks while requiring only half the tool-use budget of traditional trajectory-level RL methods.
What Problem Does This Paper Address?
Current Reinforcement Learning with Verifiable Rewards (RLVR) typically uses trajectory-level rollouts. The model generates a full sequence of thoughts and tool calls, gets a final reward (correct/incorrect), and updates its policy.
The authors identify two major flaws in this approach:
- Neglected Exploration: It fails to explore the "latent" behaviors within a multi-turn interaction. If a tool returns unexpected data, the model might need to pivot its strategy, but standard RL doesn't specifically encourage exploring those pivot points.
- The Entropy Spike: The researchers discovered that LLMs exhibit a massive spike in token entropy (uncertainty) immediately after receiving feedback from a tool. Trajectory-level RL ignores this signal, missing the opportunity to "branch out" and find better reasoning paths when the model is most confused.
Methods
ARPO introduces two core components to bridge the gap between static reasoning and dynamic agentic interaction:
1. Entropy-based Adaptive Rollout
Instead of sampling $N$ independent full trajectories, ARPO uses an "adaptive beaming" approach.
- Initial Rollout: The model starts with a few global trajectories.
- Monitoring Uncertainty: After every tool call, the algorithm calculates the entropy of the next few generated tokens.
- Dynamic Branching: If the entropy exceeds a certain threshold—meaning the tool's output has made the model uncertain—ARPO triggers partial sampling. It "branches" the current path into multiple new reasoning directions from that specific point. This allows the model to explore different ways to interpret and act on tool feedback.
2. Advantage Attribution Estimation
When you have multiple branches originating from the same root, how do you assign credit? ARPO explores two settings:
- Hard Advantage: Explicitly calculating different rewards for shared prefix tokens versus unique branched tokens.
- Soft Advantage: Leveraging the importance sampling ratios within the Group Relative Policy Optimization (GRPO) framework. The authors found that the "Soft" approach is more stable, effectively acting as a regularizer that helps the model internalize the differences in stepwise tool-use effectiveness.
Datasets
The authors evaluated ARPO across 13 challenging benchmarks categorized into three domains:
- Mathematical Reasoning: AIME 2024/2025, MATH500, MATH, and GSM8K.
- Knowledge-Intensive Reasoning: HotpotQA, 2WikiMultihopQA, MusiQue, and Bamboogle.
- Deep Search: GAIA (General AI Assistant), Humanity’s Last Exam (HLE), WebWalker, and xbench-DeepSearch.
Key Insights and Ideas
- Uncertainty as a Map: The most important "aha moment" is using token entropy as a heuristic for exploration. High entropy isn't just a sign of failure; it's a signal of a "decision point" where the model's policy is most malleable.
- Efficiency is King: Tool calls are expensive (in terms of time, API costs, and compute). By only branching when the model is "confused," ARPO avoids wasting the tool-use budget on straightforward steps where the model is already confident.
- Internalizing Feedback: By focusing on the tokens immediately following a tool call, ARPO helps the model "internalize" the environment's feedback, turning external tool results into better internal reasoning.
Key Results
ARPO consistently outperformed trajectory-level algorithms like GRPO and DAPO across various tasks.
- Superior Performance: On the GAIA benchmark, ARPO (using a Qwen3-14B backbone) achieved 43.7% pass@1 and 61.2% pass@5, significantly higher than the base model and other RL variants. Similar gains were observed on HLE and WebWalkerQA.
- Budget Efficiency: Remarkably, ARPO achieved these gains using 50% fewer tool calls than standard GRPO, as seen in the training step analysis.
- Better Diversity: Dimensionality reduction (PCA) of the rollout paths showed that ARPO produces more distinct and structured "clusters" of reasoning strategies compared to the more homogenous paths found in traditional RL.
- Robustness: The method proved effective across both Llama 3.1 and Qwen 2.5/3 backbones, showing it is a generalizable algorithmic improvement.
Limitations and Future Work
- Verifiable Rewards: Like most RLVR methods, ARPO relies on having a "ground truth" or a verifier to provide rewards. This is difficult for open-ended tasks where "correctness" is subjective.
- Computational Overhead: While it saves on tool calls, calculating real-time entropy and managing dynamic branching adds some complexity to the training pipeline.
- Future Directions: The authors suggest extending this to even more dynamic environments where tools might have side effects or where the "state" of the world changes independently of the agent's actions.
Paper Citation: Dong, G., et al. (2025). Agentic Reinforced Policy Optimization. Published as a conference paper at ICLR 2026. GitHub Repository