Efficient Context Selection for Long-Context QA: No Tuning, No Iteration, Just Adaptive‐k

Chihiro Taguchi, Seiji Maekawa, Nikita Bhutani

TL;DR

This paper introduces Adaptive-k, a novel, single-pass method for efficiently selecting relevant context for Large Language Models (LLMs) in question answering tasks. By analyzing the distribution of similarity scores between queries and candidate documents, Adaptive-k dynamically determines the optimal amount of context to retrieve, significantly reducing token usage and computational cost without requiring model fine-tuning or iterative prompting. This approach achieves competitive or superior accuracy compared to existing methods, especially for complex aggregation-style questions.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly capable, but their effectiveness in question answering (QA) is often limited by context length. Two main approaches address this: Long-Context Language Models (LCLMs), which have larger input windows, and Retrieval-Augmented Generation (RAG), which retrieves relevant external documents to inform the LLM. However, a critical challenge for both is determining how much context to provide.

  • Fixed-size retrieval (e.g., top-k documents) is suboptimal because it can either miss crucial information (too little context) or overwhelm the LLM, increasing latency and costs (too much context).
  • Existing adaptive methods often rely on iterative LLM prompting or require white-box access to the model (e.g., fine-tuning, internal KV cache access), making them incompatible with closed-source or API-based LLMs and limiting their practical deployment.
  • Furthermore, the optimal context size varies significantly depending on the query type. Factoid questions might only need a few specific facts, while aggregation questions (requiring synthesis of information from multiple sources) demand more comprehensive context.

This paper aims to solve the problem of efficiently and adaptively selecting the optimal number of context documents for a given query and LLM, without requiring model fine-tuning, iterative prompting, or special access to the LLM.

Methods

The core of the proposed method, Adaptive-k retrieval, is a single-pass approach that leverages the distribution of similarity scores between a query and candidate documents.

  1. Similarity Calculation: Given a query q and a set of candidate context documents C, the system computes embeddings for both and then calculates the cosine similarity scores between the query embedding and each document embedding.
  2. Gap Analysis: These similarity scores are sorted in descending order. The algorithm then identifies the largest "gap" or steepest drop between consecutive similarity scores.
  3. Adaptive Threshold: The index k corresponding to this largest gap is identified. This k represents the estimated boundary between relevant and irrelevant documents. Adaptive-k retrieves the top k documents before this gap.
  4. Plug-and-Play: The method is designed to be "plug-and-play," meaning it can be integrated into existing RAG pipelines without requiring model fine-tuning, changes to the LLM, or iterative prompting. It works with any retriever and LLM that accepts text input.
  5. Refinements: To mitigate potential issues, the authors add a small buffer of B=5 documents after the identified threshold k and restrict the search for the largest gap to the top 90% of documents to avoid retrieving excessively large portions of context.

The time complexity of this algorithm is O(n log n) due to sorting, which is efficient for a single-pass operation.

Datasets

The paper evaluates Adaptive-k on two types of benchmarks:

  • Factoid QA:
    • HELMET (Yen et al., 2025): This benchmark includes tasks from HotpotQA (Yang et al., 2018), Natural Questions (NQ) (Kwiatkowski et al., 2019), and TriviaQA (Joshi et al., 2017), adapted for long-context benchmarking with up to 128k tokens. A subset of 100 examples per dataset was used due to computational costs.
  • Aggregation QA:
    • HoloBench (Maekawa et al., 2025): This benchmark specifically focuses on aggregation-style queries. It allows for control over both total context size (fixed at 100k tokens) and the amount of relevant information within that context (varying levels: 5k, 10k, 25k, 50k tokens).

Key Insights and Ideas

  • The "Largest Gap" Heuristic: The core idea that the largest drop in similarity scores between sorted documents directly corresponds to the boundary between relevant and irrelevant context is a powerful, data-driven insight. This elegantly bypasses the need for explicit relevance labels or iterative LLM feedback.
  • Query-Specificity is Key: The paper strongly emphasizes that a single fixed retrieval strategy is insufficient. Adaptive-k's ability to adjust context size on a per-query basis is crucial for handling the diverse information needs of different question types (factoid vs. aggregation).
  • Efficiency Without Sacrificing Accuracy: The method demonstrates that significant reductions in token usage (up to 10x fewer tokens) can be achieved without compromising, and often improving, answer quality. This is a critical finding for making LLMs more practical and cost-effective.
  • Modularity and API Compatibility: Adaptive-k's plug-and-play nature, requiring no fine-tuning or special LLM access, makes it highly valuable for real-world applications, especially with the increasing use of closed-source LLM APIs.

Key Results

  • Aggregation QA: Adaptive-k outperforms the baseline SELF-ROUTE by up to +9 points in answer accuracy on HoloBench, especially in high-information tasks.
  • Token Reduction: Adaptive-k consistently achieves ~70% context recall while reducing token usage by 2x to 10x compared to full-context baselines.
  • Factoid QA: Adaptive-k matches or exceeds the accuracy of fixed-size retrieval with up to 99% reduction in input tokens, effectively pruning irrelevant content.
  • LLM and Embedding Model Compatibility: The method shows robust performance across five different LCLMs and three embedding models, highlighting its generalizability.
  • Context Recall: Adaptive-k consistently achieves a context recall score of approximately 70% across varying levels of relevant information, indicating it reliably retrieves a substantial portion of truly relevant chunks.

Limitations and Future Work

  • Task Applicability: The method is not directly applicable to tasks like summarization where the goal is to process the entire input holistically, as aggressive filtering might omit important information.
  • Non-Natural Language Tasks: Adaptive-k relies on semantic similarity and may not generalize well to tasks involving structured formats like JSON, where semantic embeddings might not capture relevance effectively.
  • Sensitivity to Surface-Level Variations: Typographical errors or noisy queries can distort similarity scores. The authors suggest a query standardization module as a potential solution.
  • Adversarial Inputs: The method could be vulnerable to adversarial inputs that artificially manipulate similarity scores. Robustness checks or adversarial training are suggested for future work.
  • Future Directions: The authors suggest pairing Adaptive-k with techniques like query expansion, iterative reranking, or generative feedback loops to further improve performance. They also propose exploring ensemble or runtime selection of embedding models for enhanced gains in recall and robustness.

Summary updated

Abstract

Retrieval-augmented generation (RAG) and long-context language models (LCLMs) both address context limitations of LLMs in open-domain question answering (QA). However, optimal external context to retrieve remains an open problem: fixing the retrieval size risks either wasting tokens or omitting key evidence. Existing adaptive methods like Self-RAG and Self-Route rely on iterative LLM prompting and perform well on factoid QA, but struggle with aggregation QA, where the optimal context size is both unknown and variable. We present Adaptive-$k$ retrieval, a simple and effective single-pass method that adaptively selects the number of passages based on the distribution of the similarity scores between the query and the candidate passages. It does not require model fine-tuning, extra LLM inferences or changes to existing retriever-reader pipelines. On both factoid and aggregation QA benchmarks, Adaptive-$k$ matches or outperforms fixed-$k$ baselines while using up to 10x fewer tokens than full-context input, yet still retrieves 70% of relevant passages. It improves accuracy across five LCLMs and two embedding models, highlighting that dynamically adjusting context size leads to more efficient and accurate QA.
BibTeX
@inproceedings{taguchi2025efficient,
  author = {Chihiro Taguchi and Seiji Maekawa and Nikita Bhutani},
  title = {{Efficient Context Selection for Long-Context QA: No Tuning, No Iteration, Just Adaptive‐k}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure