How Does Knowledge Selection Help Retrieval Augmented Generation?

Xiangci Li, Jessica OuyangEMNLP 2025

TL;DR

This paper investigates the impact of knowledge selection on Retrieval Augmented Generation (RAG) systems. They found that while strong generator models are less sensitive to knowledge selection, weaker models benefit significantly, especially on ambiguous tasks. The key takeaway is that prioritizing knowledge recall from the retriever is generally more impactful than sophisticated selection for improving RAG performance.

What Problem Does This Paper Address?

Retrieval Augmented Generation (RAG) is a powerful technique that enhances language models by providing them with external knowledge. While improving the retrieval of relevant documents has been extensively studied, the role of knowledge selection (i.e., filtering or reranking the retrieved documents) is less clear. Many studies focus on proposing new selection methods, but there's a lack of systematic empirical analysis on when and how much knowledge selection actually helps, especially with modern Large Language Models (LLMs). The authors hypothesize that there might be a publication bias towards positive results, and that the effectiveness of knowledge selection is highly dependent on the generator model's capability and the task's complexity.

Methods

The authors employ a systematic empirical approach using simulations to analyze the impact of knowledge selection. They control the quality of retrieved knowledge by blending "gold" (ground truth relevant) knowledge with "distractor" (irrelevant) knowledge in varying ratios. This allows them to simulate different levels of knowledge precision and recall that a retriever and selector might produce. They then evaluate the downstream generation performance of various LLM generators (GPT-4o-mini, LLaMA 3.1 8B, Mistral 7B-Instruct) under these simulated conditions. This simulation-based approach allows them to conduct hundreds of configurations, providing a more comprehensive view than typical ablation studies.

Datasets

The study uses two well-established datasets for evaluation:

  • Wizard of Wikipedia (WoW): An open-domain dialogue dataset based on Wikipedia knowledge. It's known for its relatively noisy gold knowledge annotations, making it a good testbed for analyzing the impact of less-than-perfect knowledge.
  • HotpotQA: A question-answering dataset derived from Wikipedia, designed to be less ambiguous than WoW. Its answers are strongly dependent on gold knowledge, and distractor knowledge is generally irrelevant.

Key Insights and Ideas

  • Generator Capability is Key: The performance of the generator model is the primary determinant of RAG effectiveness. Stronger generators are more robust to noisy knowledge and achieve higher performance across various knowledge settings.
  • Knowledge Recall Over Precision (for Strong Generators): For powerful LLMs, improving knowledge recall (getting more relevant information) is more critical than improving knowledge precision (filtering out irrelevant information) through selection. The gap between "full knowledge" (all retrieved) and "gold knowledge" (perfectly selected) is smaller for strong generators, indicating less room for improvement via selection.
  • Ambiguity Favors Selection: On more ambiguous tasks or datasets (like WoW), knowledge selection becomes more beneficial, especially for weaker generator models. In these cases, knowledge F1 (a combined measure of precision and recall) correlates better with generation performance.
  • Distractor Knowledge Can Harm: For datasets like HotpotQA where distractor knowledge is truly irrelevant, it can significantly degrade performance, even underperforming the "no knowledge" baseline. This highlights the importance of effective filtering when distractors are present.
  • Non-Monotonic Trends: The authors observed non-monotonic relationships between knowledge selection improvements and generation performance on WoW. This means that sometimes, increasing precision might initially hurt performance before improving it, suggesting a complex interplay.

Key Results

  • RAG is Beneficial: All tested LLMs performed significantly better with RAG compared to using only their internal knowledge, even for models pre-trained on similar data.
  • "Full Knowledge" is a Strong Baseline: Providing all retrieved knowledge (without selection) serves as a robust baseline, especially for strong generators.
  • HotpotQA Distractors are Harmful: In HotpotQA, generators receiving mostly distractor knowledge performed worse than the "no knowledge" setting.
  • WoW Distractors are Less Harmful: In WoW, "distractor" knowledge might still contribute positively to some extent, leading to less severe performance degradation.
  • Knowledge Recall Correlation: Knowledge recall showed a very strong correlation with answer F1 for strong generators (GPT-4o-mini and LLaMA 3.1 8B) on both datasets.
  • Weak Generators Benefit from Selection: For weaker generators like Mistral 7B-Instruct, knowledge F1 (precision and recall combined) showed a stronger correlation with answer F1, indicating the selector's importance.

Limitations and Future Work

  • Computational Resources: The study used a subset of the datasets and only three LLMs due to computational costs. This might mean some subtle phenomena were missed, and the contour plots in Figure 4 are not perfectly smooth.
  • Dataset Limitations: The availability of RAG datasets with human-annotated knowledge is limited. WoW, while considered noisy, is one of the cleanest available, making it hard to definitively verify hypotheses about larger solution spaces without re-annotating.
  • Uniform Sampling: The simulation used uniform sampling for knowledge selection, whereas real-world selectors might have preferences. However, the authors found that knowledge precision and recall were still good predictors of performance.
  • Future Work: The authors suggest that future practitioners should benchmark RAG systems across "no knowledge," "full knowledge," and "gold knowledge" settings to understand the potential gains from selection. They also emphasize prioritizing knowledge recall for modern LLMs and considering the impact of noisy annotations, which can lead to non-monotonic behaviors.

Summary updated

BibTeX
@inproceedings{li2025how,
  author = {Xiangci Li and Jessica Ouyang},
  title = {{How Does Knowledge Selection Help Retrieval Augmented Generation?}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure