Abstract
Retrieval-Augmented Generation (RAG) systems have emerged as a pivotal methodology for enhancing Large Language Models (LLMs) through the dynamic integration of external knowledge. To further improve RAG's flexibility, Agentic RAG introduces autonomous agents into the workflow. However, Agentic RAG faces several challenges: (1) the success of each step depends on both high-quality planning and accurate search, (2) the lack of supervision for intermediate reasoning steps, and (3) the exponentially large candidate space for planning and searching. To address these challenges, we propose DecoupleSearch, a novel framework that decouples planning and search processes using dual value models, enabling independent optimization of plan reasoning and search grounding. Our approach constructs a reasoning tree, where each node represents planning and search steps. We leverage Monte Carlo Tree Search to assess the quality of each step. During inference, Hierarchical Beam Search iteratively refines planning and search candidates with dual value models. Extensive experiments across policy models of varying parameter sizes demonstrate the effectiveness of our method.
Introduction: The Challenges of Agentic RAG
Retrieval-Augmented Generation (RAG) has become the standard for grounding Large Language Models (LLMs) in external facts. However, the field is shifting toward Agentic RAG, where the model doesn't just receive documents but actively decides what to search for and how to reason through a problem. This transition is significantly harder because a single wrong turn in a multi-step reasoning chain can lead the model into a hallucination dead end.
Current Agentic RAG systems face three major hurdles:
- Tight Coupling: If a model generates a slightly off-target plan, the search fails. If the search fails, the next plan is based on bad data. They are interdependent points of failure.
- The Supervision Gap: Most datasets only provide the final answer. We don't actually know if a model's intermediate reasoning step was "good" or just a lucky path to the right result.
- Exponential Search Space: In complex multi-hop questions, the number of possible reasoning paths grows too fast for simple greedy decoding to handle effectively.
Methodology: Decoupling Planning and Search
DecoupleSearch addresses these issues by treating planning and searching as two distinct skills, optimizing them independently to make Agentic RAG more robust and efficient.
MCTS Annotation for Intermediate Supervision
To overcome the lack of intermediate supervision, DecoupleSearch utilizes Monte Carlo Tree Search (MCTS) offline. The model explores thousands of reasoning paths for training queries. Paths that lead to the correct answer are marked as "good," while those that fail are "bad." This creates a rich "reasoning tree" that serves as the ground truth for training dual value models.
Dual Value Models and Hierarchical Beam Search
Instead of one generic reward model, DecoupleSearch trains two auxiliary "heads" on top of the LLM: a Planning Value Model ($V_\phi$) to predict the quality of a reasoning step, and a Search Value Model ($V_\psi$) to predict the utility of retrieved documents. These models share most parameters with the main policy model, ensuring computational efficiency.
During inference, the framework employs Hierarchical Beam Search (HBS). Rather than picking a single best next word, HBS samples multiple candidate plans, prunes them using the Planning Value Model, and then samples and prunes search queries for the remaining plans. This "look-ahead" mechanism allows the model to bypass dead ends that would trap a standard greedy agent.
Experimental Results and Insights
The researchers evaluated DecoupleSearch on five rigorous benchmarks, including multi-hop QA (HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle) and single-hop QA (TriviaQA). DecoupleSearch consistently outperformed vanilla RAG, iterative RAG, and existing agentic frameworks like RAgent and Search-o1.
- Performance Gains: On the Qwen2.5-7B backbone, DecoupleSearch achieved an average Exact Match (EM) score of 39.93, a significant jump from the 35.87 achieved by the next best agent (GreedyAgent).
- Complexity Handling: On MuSiQue—notoriously difficult for its complex reasoning chains—DecoupleSearch improved the EM score from roughly 12% to over 17%.
- Inference-Time Scaling: A key finding is that a smaller 7B model using DecoupleSearch's hierarchical search can outperform a much larger 14B model using standard methods, suggesting that "thinking longer" can compensate for having fewer parameters.
Conclusion
DecoupleSearch demonstrates that decoupling the "thinking" (planning) from the "looking up" (searching) creates a more resilient Agentic RAG system. While the MCTS annotation process is computationally expensive, the resulting dual-model approach is more resilient to distractor documents and significantly more effective at navigating complex multi-hop reasoning tasks. Future work may involve integrating "agent-aware" retrievers and optimizing hardware to maintain multiple active reasoning paths during inference.
Summary updated