InfoGain-RAG: Boosting Retrieval-Augmented Generation through Document Information Gain-based Reranking and Filtering

Zihan Wang, Zihan Liang, Zhou Shao, Yufei Ma, Huangyu Dai, Ben Chen, Lingtao Mao, Chenyi Lei, Yuqing Ding, Han LiEMNLP 2025

TL;DR

This paper introduces InfoGain-RAG, a novel framework that significantly improves Retrieval-Augmented Generation (RAG) by introducing a new metric, Document Information Gain (DIG), to quantify the actual contribution of retrieved documents to an LLM's answer generation. By training a specialized reranker on these DIG scores, InfoGain-RAG effectively filters out irrelevant or misleading documents and prioritizes the most valuable ones, leading to substantial accuracy gains across various LLMs and tasks.

What Problem Does This Paper Address?

Large Language Models (LLMs) often struggle with issues like hallucination and outdated knowledge. Retrieval-Augmented Generation (RAG) aims to mitigate these by providing external documents to LLMs. However, current RAG systems often fail to effectively determine which retrieved documents are truly helpful for generating accurate answers. This leads to the inclusion of irrelevant or even misleading information, which degrades the final output quality. Existing methods for improving RAG document selection either optimize the retriever (which becomes complex with multiple retrievers), rely on expensive LLM self-reflection, or use rerankers that primarily focus on semantic similarity, potentially missing documents that offer crucial information for the specific answer.

Methods

The core of InfoGain-RAG lies in two main components:

  • Document Information Gain (DIG): This novel metric quantifies a document's contribution to answer generation. It's calculated by measuring the change in the LLM's confidence in generating the correct answer with and without a specific document. A higher DIG score indicates a more valuable document. To address challenges like length bias and token importance in confidence estimation, they propose a two-step approach:
  • Sliding Window Smoothing: Mitigates length bias by averaging token probabilities within a window.
  • Token Importance Weighting: Assigns higher weights to initial tokens, which often carry more semantic weight.
  • Multi-task Reranker: A specialized reranker is trained using a multi-task strategy that combines two objectives:
  • Document Relevance Classification (Cross-Entropy Loss): Distinguishes between helpful and noisy documents based on their DIG scores (above or below certain thresholds).
  • Document Ranking Optimization (Margin Loss): Learns the relative importance of documents based on their DIG values, inspired by Circle Loss.

This combined approach allows the reranker to simultaneously learn to classify relevant documents and order them effectively, leading to robust document selection. The framework is designed to be plug-and-play and efficient, requiring only a single LLM call during inference.

Datasets

The paper uses the following datasets for training and evaluation:

  • TriviaQA: A large-scale, distantly supervised challenge dataset for reading comprehension.
  • NaturalQA: A dataset of questions and answers derived from Google's search records and Wikipedia.
  • PopQA: Focuses on long-tail entities and assesses LLM's ability to grasp infrequent factual knowledge.
  • FM2: Designed to test LLMs' ability to answer simple, factual questions.
  • December 2018 Wikipedia dump: Used as the retrieval corpus.

No new benchmarks were introduced; instead, the paper leverages existing, publicly available datasets.

Key Insights and Ideas

  • DIG as a True Measure of Utility: The paper convincingly argues that semantic similarity alone is insufficient for selecting useful documents. DIG, by directly measuring the impact on LLM confidence, provides a more accurate signal of a document's actual contribution to the answer.
  • The Power of Combined Objectives: The multi-task learning approach for the reranker, combining classification and ranking, proves more effective than using either objective alone. This suggests that a holistic understanding of document utility (both relevance and relative importance) is crucial.
  • LLM-Agnostic DIG Data: The trained reranker's performance is consistent regardless of which LLM was used to generate the DIG scores for training data. This highlights the intrinsic nature of query-document correlations captured by DIG and makes the framework more generalizable.
  • Filtering is Key: The experimental results clearly show that filtering out documents with negative DIG scores significantly boosts performance, underscoring the importance of removing misleading information.

Key Results

InfoGain-RAG demonstrates substantial improvements across various LLMs and datasets:

  • NaturalQA: Achieves significant gains in Exact Match (EM) accuracy: +17.9% over naive RAG, +4.5% over self-reflective RAG, and +12.5% over modern ranking-based RAG.
  • GPT-4: Shows an average of 15.3% increment on advanced proprietary models.
  • Single Retriever: Outperforms naive RAG, BGE-Reranker, and even the much larger proprietary GTE-7B reranker in many cases. For example, on TriviaQA, it improves Qwen2.5-72B's accuracy by 16.4% over naive RAG and 2.9% over GTE-Reranker.
  • Multiple Retrievers: Maintains consistent superiority, achieving the best performance on all four tasks.
  • Efficiency: The framework maintains low computational overhead, requiring only a single LLM call during inference and using a relatively small 355M parameter reranker.

Limitations and Future Work

  • Text Modality Only: The current implementation is tested only on text. The authors suggest it's theoretically extensible to other modalities like visual or code data.
  • Computational Constraints: The reranker is limited to 355M parameters due to computational constraints. Larger models could potentially offer better performance but might increase inference latency.
  • Factual Inaccuracies: The DIG metric, while effective, cannot distinguish factual inaccuracies within retrieved documents. An additional module might be needed to address this.
  • Future Work: The authors hope for collaborative efforts to address these limitations, particularly regarding factual accuracy and multi-modal extensions.

Summary updated

BibTeX
@inproceedings{wang2025infogain,
  author = {Zihan Wang and Zihan Liang and Zhou Shao and Yufei Ma and Huangyu Dai and Ben Chen and Lingtao Mao and Chenyi Lei and Yuqing Ding and Han Li},
  title = {{InfoGain-RAG: Boosting Retrieval-Augmented Generation through Document Information Gain-based Reranking and Filtering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure