Predicting the Benefit of Retrieval Augmentation in Open-Domain Question Answering

Or Dado, David Carmel, Oren KurlandarXiv 2026 · DOI · arXiv

TL;DR

Retrieval-Augmented Generation (RAG) is not a universal win; irrelevant or distracting context can frequently degrade an LLM's response quality. This paper introduces a framework to predict "RAG gain"—whether external context will actually help a specific query—before generating an answer. By introducing a novel "Internal Prober" that reads the LLM's internal hidden states during the initial prompt pass, the authors achieve near state-of-the-art selective RAG performance while slashing generation token compute by 82–95%.

What Problem Does This Paper Address?

While RAG has become the default architecture for grounding Large Language Models (LLMs) in external knowledge, it suffers from a major assumption: that more context is always better. In reality, retrieval mechanisms often fetch noisy, irrelevant, or contradictory passages that distract the LLM, leading to hallucinations or degraded answers. Furthermore, running retrieval and processing long contexts for every single query introduces significant latency and computational costs.

The authors address the challenge of RAG gain prediction: accurately estimating whether incorporating retrieved context will improve or harm response quality for a given question before the model spends time and compute generating a full response. Solving this enables Selective RAG, where a system dynamically decides on a per-question basis whether to use RAG or rely solely on the LLM's internal parametric knowledge.

Methods

The paper systematically evaluates traditional Query Performance Prediction (QPP) techniques and introduces several novel supervised approaches, categorized by when they execute:

  1. Pre-Retrieval Predictors: Unsupervised metrics (e.g., IDF, SCQ, VAR) that analyze only the question and corpus-level statistics.
  2. Post-Retrieval Predictors: Methods evaluating the retrieved documents. This includes unsupervised baselines (e.g., analyzing retrieval score distributions) and a new supervised model, Bert-Ret, which uses a ModernBERT-large encoder to model the semantic relationship between the query and top passages to directly predict gain.
  3. Post-Generation Predictors: Methods that look at the generated answers. The authors introduce Bert-Gen-2A, a supervised model that ingests the question, retrieved passages, and both generated answers (with and without RAG) to make a highly accurate prediction.
  4. Semi-Generation (Internal Prober): To avoid the massive computational cost of generating full answers just to predict gain, the authors design a novel white-box probing method.
Internal Prober Architecture

As shown in Figure 3, the Internal Prober uses a single frozen LLM backbone with two parallel, lightweight Low-Rank Adaptation (LoRA) modules. One branch processes the question alone ($q$), while the other processes the question with retrieved passages ($q; p_1...p_5$). By extracting and comparing the final hidden states of these prompt forward passes through linear regression heads, it estimates the semantic quality delta without generating a single output token.

Datasets

The authors evaluate their approaches across three widely used open-domain question-answering benchmarks:

  • NQ-Open (Natural Questions)
  • HotpotQA
  • TriviaQA

To ensure rigorous evaluation, they enforce a standardized partitioning strategy, sampling 24,000 questions for training supervised predictors, 6,000 for hyperparameter tuning, and 3,600 entirely distinct questions for the final test sets.

Key Insights and Ideas

  • Pre-retrieval methods are blind to LLM capabilities: Traditional search metrics (like IDF) are entirely ineffective at predicting RAG gain because they fail to account for what the LLM already knows internally.
  • Supervision is a game-changer: Explicitly training a model (Bert-Ret or Bert-Gen) to predict the delta in answer quality significantly outperforms unsupervised heuristics like token entropy or uncertainty.
  • The LLM "knows" early: The internal representations of the LLM during the initial prompt pass contain strong signals about whether the external context aligns well with the query.
  • Partial generation yields diminishing returns: Generating just a few tokens before probing the internal state rapidly closes the performance gap to full post-generation methods, saturating around $K=8$ tokens.
Internal Prober Performance vs. Tokens Generated (K)

Key Results

  • State-of-the-Art Prediction: Bert-Gen-2A achieved the highest overall prediction accuracy, posting Pearson correlation scores between predicted and actual gain of up to 0.87–0.89 on TriviaQA.
  • Efficiency Breakthrough: The Internal Prober significantly outperformed all post-retrieval methods and closed the gap to Bert-Gen-2A while reducing auto-regressive decoding computation by 82% to 95%.
  • Selective RAG Wins: When deploying these predictors in a routing framework (using RAG only if predicted gain $> 0$), the Selective RAG strategy consistently outperformed the standard "Always RAG" approach in both final answer accuracy and quality across all datasets and backbone LLMs (Falcon3-10B and Llama-3.1-8B).

Limitations and Future Work

  • White-Box Requirement: The highly efficient Internal Prober requires direct access to the LLM’s internal hidden states and activations. This makes it incompatible with commercial, closed-source "black-box" APIs (e.g., OpenAI, Anthropic) in its current form.
  • Scale Constraints: Due to hardware limitations, the study was restricted to mid-sized open models (8B–10B parameters). Scaling behavior on larger models (70B+) remains unexamined.
  • Future Directions: The authors suggest exploring multi-turn conversational QA settings and integrating broader sets of multimodal or structural predictive signals.

Summary updated

Abstract

While retrieval augmented generation has become a common approach for enhancing question answering systems, retrieval is not universally advantageous. We study the problem of predicting whether incorporating external retrieved information is likely to improve response quality for a given question. To this end, we evaluate a range of prediction methods that are based on retrieval signals, answer characteristics, and semantic consistency between generated responses and retrieved passages. We further devise a predictor that probes the LLM's internal state. Its prediction performance significantly narrows the performance gap between post-generation methods which are computationally demanding and pre-generation (post-retrieval) methods. We use the prediction methods to devise a selective retrieval framework that dynamically chooses between retrieval and non-retrieval generation modes per question. Experimental results demonstrate that selectively applying retrieval augmentation yields answer quality that transcends that of using retrieval for all queries.
BibTeX
@misc{dado2026predicting,
  author = {Or Dado and David Carmel and Oren Kurland},
  title = {{Predicting the Benefit of Retrieval Augmentation in Open-Domain Question Answering}},
  year = {2026},
  eprint = {2604.07985},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.07985},
}
Figure