Semantic Contribution-Aware Adaptive Retrieval for Black-Box Models

Qinhong Lin, Zhongliang Yang, Yuang Cai, Dingfu Yu, Xuan Xu, Yu Li, Linna ZhouEMNLP 2025

TL;DR

This paper introduces SCAAR, a novel framework for improving the accuracy of Large Language Models (LLMs) in black-box scenarios by dynamically adapting retrieval strategies. SCAAR achieves this by analyzing the semantic contribution of words to predict when and what information to retrieve, leading to more relevant and efficient knowledge augmentation.

What Problem Does This Paper Address?

Large Language Models (LLMs) are powerful tools for various NLP tasks, but they often suffer from "hallucinations" – generating factually inaccurate or inconsistent information, especially in complex reasoning tasks. Retrieval-Augmented Generation (RAG) is a common solution, where LLMs retrieve relevant information from a knowledge base before generating a response. However, conventional RAG methods often perform retrieval at fixed intervals or based on simple rules, which can be inefficient and lead to irrelevant retrieved content. Existing adaptive retrieval methods, which aim to dynamically decide when and what to retrieve, often rely on access to the LLM's internal states (e.g., hidden states or attention scores). This makes them unsuitable for "black-box" LLMs, where such internal information is inaccessible. The paper addresses this gap by proposing an adaptive retrieval method that works effectively even when only the model's output probabilities are available.

Methods

The core of SCAAR lies in its two main components:

  • Semantic Contribution-Aware Adaptive Retrieval Determination (SCW): This method determines when to retrieve. It uses a leave-one-out approach with an external cross-encoder to measure the semantic contribution of each word in a generated sentence. This contribution score is then used to dynamically adjust a retrieval threshold for each word. If a word's probability falls below its scaled threshold, retrieval is triggered. This approach is designed to be lightweight and generalizable to black-box models.
  • Percentile-Filtered Query (PFQ) Construction: This method determines what to retrieve. After retrieval is triggered, SCAAR identifies the most semantically significant words in the potentially hallucinated sentence (based on their contribution scores) and retains the top 'a%' of these words. These high-contribution words, along with the original question, are used to construct a more focused retrieval query. This helps filter out less relevant information and ensures the retrieved context is more pertinent to the model's knowledge gap.

Datasets

The paper evaluates SCAAR on four widely-used knowledge-intensive datasets:

  • 2WikiMultiHopQA
  • HotpotQA
  • IIRC
  • StrategyQA

These datasets are designed to test complex reasoning and information retrieval capabilities.

Key Insights and Ideas

  • Black-Box Adaptability is Crucial: The paper highlights the significant limitation of existing adaptive retrieval methods that require internal model states, making them unusable for many powerful LLMs. SCAAR's success in black-box settings is a key contribution.
  • Semantic Contribution as a Proxy for Knowledge Gaps: The core idea that a word's semantic contribution can indicate a model's uncertainty or knowledge gap is powerful. By quantifying this, SCAAR can intelligently decide when to seek external information.
  • Dynamic Thresholding Outperforms Static: The experiments consistently show that adaptive methods, particularly SCAAR, outperform static retrieval approaches (like fixed-interval retrieval) in both performance and efficiency.
  • Query Formulation Matters: The PFQ method, by focusing on semantically important words, leads to more targeted and effective retrieval queries compared to simpler methods.

Key Results

  • SCAAR achieved the highest scores on each of the four datasets when evaluated with GPT-40, outperforming baseline methods like FLARE and DRAGIN.
  • Across various LLMs (Llama-2, Llama-3, GPT-40), SCAAR consistently demonstrated superior or competitive performance compared to adaptive white-box retrieval methods and static retrieval approaches.
  • The retrieval efficiency (Seff) of SCAAR was also competitive, indicating that it avoids unnecessary retrieval overhead.
  • Ablation studies confirmed the effectiveness of both the semantic contribution weighting (SCW) and the percentile-filtered query construction (PFQ) components of SCAAR.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Enhancing Semantic Weight Representativeness: While DeBERTa showed better performance than RoBERTa as the semantic encoder, domain-specific fine-tuning of the encoder could further improve weight representativeness.
  • Learnable Quantile Filtering: The current percentile filtering relies on heuristic constants. Training a classifier for percentile prediction is suggested as a necessary next step.
  • Optimizing Dense Passage Retrieval: Improving the ability to capture semantics from short upcoming sentences in dense passage retrieval remains a challenge.
  • LLM-based Metrics: While EM and F1 scores were used, incorporating LLM-based metrics could provide a more comprehensive evaluation of retrieval accuracy.

Summary updated

BibTeX
@inproceedings{lin2025semantic,
  author = {Qinhong Lin and Zhongliang Yang and Yuang Cai and Dingfu Yu and Xuan Xu and Yu Li and Linna Zhou},
  title = {{Semantic Contribution-Aware Adaptive Retrieval for Black-Box Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure