TL;DR
This paper introduces RLTR, a novel reinforcement learning framework that decouples LLM agent training to focus solely on optimizing the agent's planning capabilities. By using a "tool-use completeness" reward, RLTR bypasses the need for verifiable final answers, a major bottleneck in real-world scenarios, leading to significantly improved planning performance and, consequently, better overall agent responses.
What Problem Does This Paper Address?
Large Language Model (LLM) agents are powerful tools for tasks like code generation and question answering. Their effectiveness hinges on two core capabilities: action planning (deciding which tools to use and in what order) and answer summarization (synthesizing information into a final response). Current training methods often try to optimize both simultaneously using end-to-end reinforcement learning (RL). However, this approach faces two critical challenges:
- Scarcity of Verifiable Data and Reward Hacking: In real-world industrial scenarios, data with definitively correct answers is rare and expensive to obtain. Existing methods often rely on reward models trained on final answers, which are prone to "reward hacking" – where the agent learns to produce outputs that trick the reward model without actually solving the problem. This leads to inaccurate rewards and suboptimal training.
- Conflicting Objectives and Credit Assignment Difficulty: Optimizing both planning and summarization simultaneously creates conflicting objectives. The RL reward, derived from the final answer, makes it hard to assign credit or blame to specific planning steps that occurred earlier in the process. This "credit assignment problem" hinders the effective improvement of the planning module.
Methods
The paper proposes Reinforcement Learning with Tool-use Rewards (RLTR), a framework that addresses these challenges by:
- Decoupling Training: Instead of end-to-end optimization, RLTR focuses exclusively on optimizing the Planner module. The Summarizer module can be any LLM, even an untrained one, and is used after the Planner has done its job.
- Tool-Use Completeness Reward: The core innovation is a new reward signal that evaluates the completeness of the tool invocation sequence. This is achieved using a "Comp. Checker" LLM that verifies if the agent has successfully used the necessary tools to gather all required information. This reward is high if the planning process is complete and low otherwise.
- Cold-Start Initialization: The Planner is initially trained using knowledge distillation from a teacher LLM and rejection sampling to ensure it can handle planning tasks effectively.
- Multi-Turn Reinforcement Learning: The Planner is then optimized using the tool-use completeness reward in a multi-turn RL setting. Rule-based rewards (e.g., for avoiding repetitive tool calls) are also incorporated to stabilize training.
Datasets
- Industry Dataset: A proprietary dataset collected by the authors, consisting of approximately 4k training and 0.5k test samples, designed for training and evaluating agents on search and code tools in Chinese. The test set includes "normal" and "hard" levels to challenge performance on difficult queries.
- Open-source Dataset: ChineseSimpleQA (He et al., 2024) was used as an additional test set. Samples that could be answered without tool invocation were excluded to ensure a challenging evaluation.
Key Insights and Ideas
- Focusing on Process Over Outcome: The most significant insight is that optimizing the process (how the agent plans and uses tools) can be more effective and robust than solely optimizing for the outcome (the final answer), especially when verifiable outcome data is scarce.
- Tool-Use Completeness as a High-Fidelity Reward: The tool-use completeness reward provides a more direct and reliable signal for the Planner than a reward based on the final answer's correctness. This is because it directly assesses the quality of the agent's actions, bypassing the complexities and potential inaccuracies of evaluating the final synthesized response.
- Decoupling Simplifies Optimization: By separating the planning and summarization stages, RLTR avoids the conflicting objectives and credit assignment issues inherent in end-to-end training, leading to more stable and effective optimization of the Planner.
- Improved Planning Boosts Summarization: Even without dedicated training for the Summarizer, an optimized Planner provides more comprehensive and accurate information, which in turn leads to better summarization and a higher quality final response.
Key Results
- Planning Performance: RLTR achieved an 8%-12% improvement in planning performance (measured by tool-use completeness) compared to end-to-end baselines on both SFT and RL stages. This improvement was particularly pronounced on the "hard" subset of the industry dataset.
- Final Response Quality: This enhanced planning capability translated to a 5%-6% increase in the overall agent's final response quality (measured by accuracy, helpfulness, and relevance) across various models.
- Training Stability: The tool-use completeness reward was found to be more accurate and led to more stable training compared to answer-based rewards, as evidenced by higher accuracy and F1 scores on manually labeled samples.
- Robustness: RLTR demonstrated robust performance across different RL algorithms (PPO, GRPO, REINFORCE++).
Limitations and Future Work
- Language Specificity: The current deployment scenario and experiments are primarily focused on Chinese. The authors acknowledge the need to construct agent datasets that include more languages to enhance the approach's effectiveness across different linguistic contexts.
- Summarizer Optimization: The work primarily focuses on optimizing the Planner. The authors employed an untrained LLM as the Summarizer and did not specifically investigate optimization strategies for this component. Future work plans to explore methods for optimizing the Summarizer to further improve the overall agent system performance.
Summary updated