Supportiveness-based Knowledge Rewriting for Retrieval-augmented Language Modeling

Zile Qiao, Wei Ye, Yong Jiang, Tong Mo, Pengjun Xie, Weiping Li, Fei Huang, Shikun Zhang

Abstract

Retrieval-augmented language models (RALMs) have recently shown great potential in mitigating the limitations of implicit knowledge in LLMs, such as untimely updating of the latest expertise and unreliable retention of long-tail knowledge. However, since the external knowledge base, as well as the retriever, can not guarantee reliability, potentially leading to the knowledge retrieved not being helpful or even misleading for LLM generation. In this paper, we introduce Supportiveness-based Knowledge Rewriting (SKR), a robust and pluggable knowledge rewriter inherently optimized for LLM generation. Specifically, we introduce the novel concept of “supportiveness”—which represents how effectively a knowledge piece facilitates downstream tasks. Based on supportiveness, we first design a training data curation strategy for our rewriter model, effectively identifying and filtering out poor or irrelevant rewrites to improve data efficacy. We then introduce the direct preference optimization (DPO) algorithm to align the generated rewrites to optimal supportiveness, guiding the rewriter model to summarize augmented content that better improves the final response. Comprehensive evaluations across six popular knowledge-intensive tasks and four LLMs have demonstrated the effectiveness and superiority of SKR. With only 7B parameters, SKR has shown better knowledge rewriting capability over GPT-4.

Introduction

While Retrieval-Augmented Language Models (RALMs) help mitigate hallucinations and outdated knowledge, they are highly dependent on the quality of the retriever. If a retriever returns noisy, irrelevant, or misleading "knowledge chunks," the performance of the downstream Large Language Model (LLM) can degrade compared to using no retrieval at all.

Existing knowledge rewriters attempt to clean this data, often using changes in model perplexity as a signal for quality. However, these methods suffer from Internal Knowledge Bias: if an LLM already knows the answer, adding helpful knowledge won't significantly change its perplexity, leading to incorrect labeling of useful information as "useless." Supportiveness-based Knowledge Rewriting (SKR) addresses this by decoupling information quality from the model's pre-existing internal knowledge.

Methodology

The SKR framework employs a two-stage approach to build a robust rewriter optimized for LLM generation.

Supportiveness Metric

The authors define "supportiveness" ($ss$) as the ratio of perplexity (without vs. with retrieval) normalized by the model's baseline confidence. This metric ensures that knowledge is valued based on its ability to help the model solve queries it previously could not.

Supportiveness-based Rewrite Data Generation (SRDG)

Candidate rewrites are generated using GPT-4 for a large training set. These are then scored by a "white-box" LLM (Mistral-7B) using the $ss$ metric. Rewrites are categorized into three groups: Successful (helpful), Failed (worse than raw data), or Irrelevant (the retriever provided nothing useful).

Supportiveness-guided Alignment (SA)

The rewriter is first fine-tuned via Supervised Fine-Tuning (SFT) on "Successful" and "Irrelevant" samples. In irrelevant cases, the model learns to output the specific token "irrelevant." Finally, Direct Preference Optimization (DPO) is used to align the rewriter to prefer higher supportiveness scores.

The SKR framework: Supportiveness-based Rewrite Data Generation and Supportiveness-guided Alignment.

Experimental Setup

SKR was trained on a diverse mix of 10 datasets covering Open-Domain QA, Reading Comprehension, and Reasoning. Evaluation was conducted across six standard benchmarks: MMLU, Natural Questions (NQ), TriviaQA (TQA), HotpotQA (HoPo), WebQuestions (WQ), and zsRE. The retrieval corpus used was a Wikipedia 2018 dump.

Results and Discussion

SKR-7B consistently outperforms larger general-purpose models like GPT-4 and GPT-3.5 in rewriting tasks.

Main results of SKR compared to various baselines across six benchmarks.

Performance and Efficiency

As shown in the experimental results, SKR-7B achieved an average improvement of 10.01 points in Exact Match (EM) scores for Llama-2-7B compared to a standard Top-5 retrieval baseline. Beyond accuracy, SKR provides significant context compression, achieving over 7.5x compression on NQ and TriviaQA. This reduces the token overhead for the downstream LLM while maintaining high signal.

Robustness and Generalization

SKR demonstrates strong robustness to noise. In scenarios with "bad" or "random" retrieval, SKR mitigates performance drops that typically cripple raw LLMs. Furthermore, the model shows impressive zero-shot generalization on unseen datasets like MMLU, outperforming specialized methods like RECOMP and FILCO.

Conclusion

SKR introduces a novel "supportiveness" metric that effectively identifies helpful knowledge relative to a model's internal state. By learning to explicitly flag irrelevant data and optimizing for supportiveness via DPO, the 7B-parameter SKR model provides a pluggable, efficient, and high-performance solution for RALMs. Future work will focus on ensuring strict faithfulness to source text to further minimize hallucination risks in abstractive rewriting.

Summary updated

Abstract

Retrieval-augmented language models (RALMs) have recently shown great potential in mitigating the limitations of implicit knowledge in LLMs, such as untimely updating of the latest expertise and unreliable retention of long-tail knowledge. However, since the external knowledge base, as well as the retriever, can not guarantee reliability, potentially leading to the knowledge retrieved not being helpful or even misleading for LLM generation. In this paper, we introduce Supportiveness-based Knowledge Rewriting (SKR), a robust and pluggable knowledge rewriter inherently optimized for LLM generation. Specifically, we introduce the novel concept of “supportiveness”—which represents how effectively a knowledge piece facilitates downstream tasks. Based on supportiveness, we first design a training data curation strategy for our rewriter model, effectively identifying and filtering out poor or irrelevant rewrites to improve data efficacy. We then introduce the direct preference optimization (DPO) algorithm to align the generated rewrites to optimal supportiveness, guiding the rewriter model to summarize augmented content that better improves the final response. Comprehensive evaluations across six popular knowledge-intensive tasks and four LLMs have demonstrated the effectiveness and superiority of SKR. With only 7B parameters, SKR has shown better knowledge rewriting capability over GPT-4.
BibTeX
@inproceedings{qiao2025supportiveness,
  author = {Zile Qiao and Wei Ye and Yong Jiang and Tong Mo and Pengjun Xie and Weiping Li and Fei Huang and Shikun Zhang},
  title = {{Supportiveness-based Knowledge Rewriting for Retrieval-augmented Language Modeling}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  url = {https://openreview.net/forum?id=uLy476ydll},
}
Figure