Abstract
Retrieval-augmented generation (RAG) and long-context language models (LCLMs) both address context limitations of LLMs in open-domain question answering (QA). However, optimal external context to retrieve remains an open problem: fixing the retrieval size risks either wasting tokens or omitting key evidence. Existing adaptive methods like Self-RAG and Self-Route rely on iterative LLM prompting and perform well on factoid QA, but struggle with aggregation QA, where the optimal context size is both unknown and variable. We present Adaptive-$k$ retrieval, a simple and effective single-pass method that adaptively selects the number of passages based on the distribution of the similarity scores between the query and the candidate passages. It does not require model fine-tuning, extra LLM inferences or changes to existing retriever-reader pipelines. On both factoid and aggregation QA benchmarks, Adaptive-$k$ matches or outperforms fixed-$k$ baselines while using up to 10x fewer tokens than full-context input, yet still retrieves 70% of relevant passages. It improves accuracy across five LCLMs and two embedding models, highlighting that dynamically adjusting context size leads to more efficient and accurate QA.
Introduction
In Retrieval-Augmented Generation (RAG), selecting the right amount of context is a "Goldilocks" problem. Providing too little context causes the model to miss critical evidence, while providing too much context leads to the "lost in the middle" phenomenon, increased latency, and higher API costs. Standard RAG systems typically use a "fixed-$k$" approach, retrieving a static number of top documents regardless of query complexity. While a simple factoid question might require only one document, complex "aggregation" questions (e.g., "List all colleges in California with over 10,000 students") may require dozens.
Existing adaptive solutions often involve heavy overhead, such as fine-tuning the LLM (e.g., Self-RAG) or iterative prompting (e.g., Self-Route), which is slow and expensive. This paper introduces Adaptive-$k$, a "plug-and-play" method that works with any off-the-shelf LLM and retriever without extra overhead.
The Adaptive-$k$ Method
The core innovation is the Largest Similarity Gap algorithm. Instead of picking a pre-defined number of documents, the system analyzes the mathematical distribution of the retriever's confidence scores in a single pass.
The process follows these steps:
- Embed & Score: Generate similarity scores between the query and all candidate documents.
- Sort: Rank documents by score in descending order.
- Find the Gap: Calculate the difference between adjacent scores. The "cutoff" point is set where the largest drop in similarity occurs.
- Add a Buffer: To ensure robustness, the system retrieves the top $k$ documents (the gap index) plus a small fixed buffer (e.g., $B=5$).
- Constraint: To prevent runaway costs, the search for the "largest gap" is restricted to the top 90% of documents.
Experimental Evaluation
The authors evaluated Adaptive-$k$ across two distinct types of challenges:
- Factoid QA: Using the HELMET benchmark (including HotpotQA, Natural Questions, and TriviaQA), featuring contexts up to 128k tokens.
- Aggregation QA: Using HoloBench, designed for queries requiring reasoning over multiple pieces of evidence spread across massive datasets.
A key insight is that the similarity distribution itself serves as a signal. For simple questions, scores drop sharply after relevant documents; for complex queries, scores remain high longer. Adaptive-$k$ naturally scales its "appetite" based on the query's inherent complexity.
Key Results
Adaptive-$k$ demonstrates that efficiency can drive accuracy. By pruning irrelevant "noise," the method helps the LLM focus on the "signal," avoiding performance degradation seen in overwhelmed long-context models.
- Token Savings: On factoid QA, Adaptive-$k$ achieved a 99% reduction in input tokens compared to full-context methods with no loss in accuracy.
- Superior Aggregation: On HoloBench, it outperformed the previous state-of-the-art modular method (SELF-ROUTE) by up to 9 points.
- Consistent Recall: The method consistently retrieved approximately 70% of relevant passages across tasks.
- Universal Compatibility: Gains were consistent across five different LLMs (including GPT-4o, Gemini 2.5 Flash, and Llama 4) and multiple embedding models.
Limitations and Future Work
While highly effective for QA, Adaptive-$k$ is not intended for summarization tasks where the goal is to process the entire input. It also relies on semantic similarity, which may be less effective for structured data like code. Future work could address sensitivity to "noisy" embedding scores or adversarial "distractor" documents designed to trick the gap detection algorithm.
Summary updated