Stop-RAG: Value-Based Retrieval Control for Iterative RAG

Jaewan Park, Solbee Cho, Jay-Yoon Lee

TL;DR

Iterative Retrieval-Augmented Generation (RAG) can be inefficient due to unnecessary retrieval steps. This paper introduces Stop-RAG, a novel value-based controller that uses a Markov Decision Process (MDP) framework to intelligently decide when to stop retrieving, leading to improved accuracy and efficiency in multi-hop question answering.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly used in agentic systems that can interact with external tools, including iterative retrieval. Retrieval-Augmented Generation (RAG) is a prime example, where LLMs repeatedly retrieve information to answer complex, multi-hop questions. However, each retrieval loop adds latency, cost, and the risk of introducing irrelevant "distractor" information. Current methods for deciding when to stop these iterative RAG processes are suboptimal: some use a fixed number of iterations, while others rely on LLM self-assessment or confidence scores, which don't always accurately predict whether further retrieval will actually be beneficial. This leads to either incomplete answers or unnecessarily long and costly query sequences. The paper addresses this gap by proposing an adaptive stopping strategy that truly learns when to stop.

Methods

The core innovation of Stop-RAG is framing the iterative RAG process as a finite-horizon Markov Decision Process (MDP). In this MDP, the "state" is the history of retrieved documents and intermediate answers, and the "actions" are either to STOP the retrieval process or CONTINUE with another iteration.

To learn an effective stopping policy, the authors employ a value-based reinforcement learning approach, specifically using a Q-learning variant with "full-width forward-view Q($\lambda$)" targets. This method allows the model to estimate the future value of continuing retrieval, not just the immediate benefit. By considering the long-term consequences of stopping or continuing, Stop-RAG can make more informed decisions. The Q-network is trained offline on pre-collected trajectories of iterative RAG, making it compatible with black-box LLMs and existing RAG pipelines without requiring online interaction during training.

Datasets

The paper evaluates Stop-RAG on three multi-hop question-answering benchmarks:

  • MuSiQue: A dataset requiring reasoning across multiple retrieval steps.
  • HotpotQA: Another multi-hop QA dataset.
  • 2WikiMultihopQA: A dataset also designed for multi-hop question answering.

For training, the authors use the official "train" partitions of these datasets. They also sample 1000 questions each from the "dev" partitions for validation and testing. The retrieval corpora for MuSiQue and 2WikiMultihopQA were constructed by aggregating context from the respective datasets, while the HotpotQA corpus used the authors' provided Wikipedia data.

Key Insights and Ideas

  • Adaptive stopping is a critical missing component: The paper argues that current agentic systems, particularly iterative RAG, lack a principled way to decide when to stop retrieving. This is a fundamental limitation that Stop-RAG aims to address.
  • Value-based control for stopping: Instead of relying on heuristics or simple confidence scores, Stop-RAG learns to predict the long-term value of continuing retrieval. This forward-looking perspective is key to making better decisions.
  • MDP formulation is powerful: Casting iterative RAG as an MDP provides a structured framework for learning optimal stopping policies.
  • Offline training is practical: Training Stop-RAG on pre-collected trajectories makes it easily deployable with existing, potentially black-box, RAG systems.

Key Results

Stop-RAG consistently outperformed baseline methods on the evaluated multi-hop QA benchmarks.

  • On their own pipeline: Stop-RAG achieved higher performance (EM and F1 scores) compared to both fixed-iteration baselines and LLM-prompted stopping (LLM-Stop). For example, on MuSiQue, Stop-RAG achieved an F1 score of 47.0, compared to 44.8 for their pipeline with LLM-Stop.
  • As a plug-and-play module: When applied to the CoRAG baseline, Stop-RAG also improved performance, demonstrating its generalizability across different RAG architectures.
  • Retrieval efficiency: Stop-RAG executed slightly more retrieval steps on average than LLM-Stop, which led to higher recall and better overall performance, indicating a better balance between retrieval depth and precision.

Limitations and Future Work

  • HotpotQA performance: On HotpotQA, Stop-RAG slightly underperformed baselines. The authors hypothesize this is because CoRAG's generator was fine-tuned for a fixed number of iterations, making it more robust to distractors, and on the challenging HotpotQA corpus, the benefit of continuing retrieval outweighed the cost of distractors.
  • Future work: The authors suggest exploring Stop-RAG with other high-performing iterative RAG frameworks.

Summary updated

Abstract

Iterative retrieval-augmented generation (RAG) enables large language models to answer complex multi-hop questions, but each additional loop increases latency, costs, and the risk of introducing distracting evidence, motivating the need for an efficient stopping strategy. Existing methods either use a predetermined number of iterations or rely on confidence proxies that poorly reflect whether more retrieval will actually help. We cast iterative RAG as a finite-horizon Markov decision process and introduce Stop-RAG, a value-based controller that adaptively decides when to stop retrieving. Trained with full-width forward-view Q($\lambda$) targets from complete trajectories, Stop-RAG learns effective stopping policies while remaining compatible with black-box APIs and existing pipelines. On multi-hop question-answering benchmarks, Stop-RAG consistently outperforms both fixed-iteration baselines and prompting-based stopping with LLMs. These results highlight adaptive stopping as a key missing component in current agentic systems, and demonstrate that value-based control can improve the accuracy of RAG systems.
BibTeX
@inproceedings{park2025stop,
  author = {Jaewan Park and Solbee Cho and Jay-Yoon Lee},
  title = {{Stop-RAG: Value-Based Retrieval Control for Iterative RAG}},
  booktitle = {MTI-LLM @ NeurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=fN4PJNYgs3},
}
Figure