Abstract
Retrieval-augmented language models (RALMs) have recently shown great potential in mitigating the limitations of implicit knowledge in LLMs, such as untimely updating of the latest expertise and unreliable retention of long-tail knowledge. However, since the external knowledge base, as well as the retriever, can not guarantee reliability, potentially leading to the knowledge retrieved not being helpful or even misleading for LLM generation. In this paper, we introduce Supportiveness-based Knowledge Rewriting (SKR), a robust and pluggable knowledge rewriter inherently optimized for LLM generation. Specifically, we introduce the novel concept of “supportiveness”—which represents how effectively a knowledge piece facilitates downstream tasks. Based on supportiveness, we first design a training data curation strategy for our rewriter model, effectively identifying and filtering out poor or irrelevant rewrites to improve data efficacy. We then introduce the direct preference optimization (DPO) algorithm to align the generated rewrites to optimal supportiveness, guiding the rewriter model to summarize augmented content that better improves the final response. Comprehensive evaluations across six popular knowledge-intensive tasks and four LLMs have demonstrated the effectiveness and superiority of SKR. With only 7B parameters, SKR has shown better knowledge rewriting capability over GPT-4.
Introduction
While Retrieval-Augmented Language Models (RALMs) help mitigate hallucinations and outdated knowledge, they are highly dependent on the quality of the retriever. If a retriever returns noisy, irrelevant, or misleading "knowledge chunks," the performance of the downstream Large Language Model (LLM) can degrade compared to using no retrieval at all.
Existing knowledge rewriters attempt to clean this data, often using changes in model perplexity as a signal for quality. However, these methods suffer from Internal Knowledge Bias: if an LLM already knows the answer, adding helpful knowledge won't significantly change its perplexity, leading to incorrect labeling of useful information as "useless." Supportiveness-based Knowledge Rewriting (SKR) addresses this by decoupling information quality from the model's pre-existing internal knowledge.
Methodology
The SKR framework employs a two-stage approach to build a robust rewriter optimized for LLM generation.
Supportiveness Metric
The authors define "supportiveness" ($ss$) as the ratio of perplexity (without vs. with retrieval) normalized by the model's baseline confidence. This metric ensures that knowledge is valued based on its ability to help the model solve queries it previously could not.
Supportiveness-based Rewrite Data Generation (SRDG)
Candidate rewrites are generated using GPT-4 for a large training set. These are then scored by a "white-box" LLM (Mistral-7B) using the $ss$ metric. Rewrites are categorized into three groups: Successful (helpful), Failed (worse than raw data), or Irrelevant (the retriever provided nothing useful).
Supportiveness-guided Alignment (SA)
The rewriter is first fine-tuned via Supervised Fine-Tuning (SFT) on "Successful" and "Irrelevant" samples. In irrelevant cases, the model learns to output the specific token "irrelevant." Finally, Direct Preference Optimization (DPO) is used to align the rewriter to prefer higher supportiveness scores.
Experimental Setup
SKR was trained on a diverse mix of 10 datasets covering Open-Domain QA, Reading Comprehension, and Reasoning. Evaluation was conducted across six standard benchmarks: MMLU, Natural Questions (NQ), TriviaQA (TQA), HotpotQA (HoPo), WebQuestions (WQ), and zsRE. The retrieval corpus used was a Wikipedia 2018 dump.
Results and Discussion
SKR-7B consistently outperforms larger general-purpose models like GPT-4 and GPT-3.5 in rewriting tasks.
Performance and Efficiency
As shown in the experimental results, SKR-7B achieved an average improvement of 10.01 points in Exact Match (EM) scores for Llama-2-7B compared to a standard Top-5 retrieval baseline. Beyond accuracy, SKR provides significant context compression, achieving over 7.5x compression on NQ and TriviaQA. This reduces the token overhead for the downstream LLM while maintaining high signal.
Robustness and Generalization
SKR demonstrates strong robustness to noise. In scenarios with "bad" or "random" retrieval, SKR mitigates performance drops that typically cripple raw LLMs. Furthermore, the model shows impressive zero-shot generalization on unseen datasets like MMLU, outperforming specialized methods like RECOMP and FILCO.
Conclusion
SKR introduces a novel "supportiveness" metric that effectively identifies helpful knowledge relative to a model's internal state. By learning to explicitly flag irrelevant data and optimizing for supportiveness via DPO, the 7B-parameter SKR model provides a pluggable, efficient, and high-performance solution for RALMs. Future work will focus on ensuring strict faithfulness to source text to further minimize hallucination risks in abstractive rewriting.
Summary updated