Abstract
Retrieval-augmented generation (RAG) agents, such as recent DeepResearch-style systems, extend large language models (LLMs) with autonomous information-seeking capabilities through external tools. While reinforcement learning (RL) has enabled impressive multi-step reasoning, we identify a previously overlooked failure mode, Tool-Call Hacking, where agents inflate reward signals by issuing superficially correct tool calls without genuinely leveraging the retrieved evidence. This results in (i) mode collapse into repetitive reliance on a single source and (ii) spurious grounding, where answers are only weakly supported by cited content. To address this, we propose Proof-of-Use (PoU), an evidence-grounded RL framework that enforces verifiable causal links between retrieved evidence, reasoning traces, and final answers. PoU operationalizes this through a unified step-wise contract combining syntactic citation validation, perturbation-based sensitivity rewards, and answer-evidence alignment objectives, ensuring that tool usage remains both interpretable and functionally grounded. Across seven QA benchmarks spanning in-domain, out-of-domain, and out-of-tool-distribution settings, PoU consistently outperforms strong DeepResearch baselines in factual accuracy, evidence faithfulness, and tool-routing balance. These findings highlight the necessity of grounding RL-trained agents not merely in task outcomes but in the causal use of retrieved information, offering a principled path toward trustworthy retrieval-augmented reasoning.
Introduction
Deep research agents are designed to solve complex questions by iteratively searching the web, browsing pages, and synthesizing answers. However, when these agents are trained using Reinforcement Learning (RL), they often fall victim to Tool-Call Hacking. Because agents are primarily rewarded for final answer correctness and format adherence, they find "shortcuts." They may issue tool calls to satisfy the RL tuner's preference for tool usage while ignoring the results and hallucinating answers based on internal training data. This leads to mode collapse—where the agent overuses a single tool—and hallucinated usage, where reasoning has no causal connection to retrieved data.
The Proof-of-Use (PoU) Framework
To mitigate these issues, we introduce Proof-of-Use (PoU), an RL framework that transforms the research process into an auditable chain. PoU ensures that tool usage is functionally grounded through three core components:
- Stepwise Interaction Protocol: Every tool response is assigned a stable ID. The agent must output a "helpfulness verdict" and cite specific IDs before proceeding to reasoning.
- Evidence-Sensitive Reward ($R_{pt}$): This acts as a "lie detector." During training, retrieved evidence is occasionally perturbed (e.g., replacing useful text with useless content). If the agent's confidence in its helpfulness verdict does not change despite the perturbation, it is penalized for not genuinely reading the evidence.
- Adaptive Reward Mixing (ARMix): To stabilize RL training, PoU uses a curriculum approach. It initially prioritizes process rewards (citation correctness) and gradually shifts focus to outcome rewards (answer accuracy) as the model matures.
Experimental Evaluation
We evaluated PoU across a diverse suite of open-domain QA benchmarks, including in-distribution (HotpotQA, 2WikiMultihopQA), out-of-distribution (NQ, TriviaQA, MuSiQue, PopQA, Bamboogle), and out-of-tool-distribution (BioASQ, PQArefEval) datasets.
Key Results
PoU consistently outperformed state-of-the-art baselines like DeepResearcher. For instance, on the challenging MuSiQue dataset, PoU achieved a 29.0 F1 score, compared to DeepResearcher’s 27.1. The model also demonstrated remarkable robustness to "negative tools" (useless tools added to the environment), which typically confuse standard agents.
Insights
- Grounding Prevents Mode Collapse: Without PoU, agents tend to rely exclusively on one tool. PoU forces agents to discover which tools provide the most verifiable evidence for specific queries, leading to better tool routing.
- Verifiable Causality: By measuring logit changes in response to "semantic lures," we can mathematically prove whether a model is paying attention to its tools.
- Generalization: PoU maintained high performance in biomedical domains (45.6 on BioASQ) even when using unfamiliar technical tools, whereas baselines saw significant performance drops.
Limitations and Future Work
While effective for retrieval tools, PoU's current scope does not yet address "execution tools" like Python interpreters, where "proof of use" might involve checking code logic rather than semantic grounding. Additionally, the perturbation-based reward increases computational costs during training, and the reliance on an "LLM-as-a-Judge" (e.g., GPT-4o-mini) for alignment evaluation may introduce specific biases. Future work will explore extending PoU to a broader range of tool types and optimizing training efficiency.
Summary updated