Do You Know What You Are Talking About? Characterizing Query-Knowledge Relevance For Reliable Retrieval Augmented Generation

Zhuohang Li, Jiaxin Zhang, Chao Yan, Kamalika Das, Kumar Sricharan, Murat Kantarcioglu, Bradley A. Malin

Abstract

Retrieval-Augmented Generation (RAG) systems often fail when user queries fall outside the scope of their internal knowledge base, leading to hallucinations triggered by irrelevant retrieved documents. This paper introduces a statistical framework using Goodness-of-Fit (GoF) tests to characterize query-knowledge relevance. By mapping high-dimensional embeddings to a one-dimensional test statistic, the proposed method identifies out-of-knowledge (OoK) queries in real-time and detects distribution shifts in user queries over time.

Introduction

The reliability of RAG systems depends on the relevance of retrieved context. When a system encounters an "out-of-knowledge" (OoK) query—one that cannot be answered by the provided corpus—it typically retrieves the "least irrelevant" documents. These distractors often cause Large Language Models (LLMs) to generate confident but false answers. Current systems lack efficient mechanisms to flag these queries online or to detect when the underlying knowledge base has become obsolete (offline).

Methodology

The authors propose treating query relevance as a statistical hypothesis testing problem. The core approach involves mapping text embeddings into a one-dimensional test statistic that measures how well a query fits the "In-Knowledge" (IK) distribution.

Online and Offline Testing Scenarios

The framework addresses two primary scenarios:

  1. Online Testing: A Goodness-of-Fit (GoF) test is used to evaluate individual queries in real-time. A query is rejected as OoK if its test statistic is a statistical outlier relative to a reference IK distribution.
  2. Offline Testing: To detect long-term distribution shifts, the two-sample Kolmogorov-Smirnov (KS) test compares a batch of recent queries against historical data.
Overview of the statistical framework for identifying out-of-knowledge queries and detecting distribution shifts.

To handle cases where a set of true IK queries is unavailable, the authors utilize LLMs to generate synthetic in-knowledge queries from corpus text chunks. These synthetic queries serve as a proxy for the system's knowledge boundaries, making the approach "training-free."

Experimental Setup

The framework was evaluated using eight QA datasets and two specialized medical corpora: a collection of 18 medical Textbooks and the PubMed abstract database. Ground truth benchmarks included MedQA-US and PubMedQA.

Key Results and Insights

The study yielded several critical insights into the behavior of RAG systems and the effectiveness of statistical relevance testing.

Detection Performance

The GoF test, particularly when using an Energy-based score, demonstrated high accuracy in distinguishing IK from OoK queries, often achieving an AUROC between 0.95 and 0.99. The approach is also highly sample-efficient, maintaining performance even with limited reference data.

LLMs as Relevance Judges

A striking finding was that GPT-3.5 and GPT-4 performed poorly at identifying OoK queries. These models often assigned high relevance scores to completely irrelevant context, highlighting the need for external statistical validation.

Distributional Separation

As shown in the frequency distributions, synthetic queries effectively mirror the distribution of real IK queries. This allows for clear thresholding to reject irrelevant inputs, as OoK queries (e.g., from TruthfulQA) form a distinct, non-overlapping distribution.

Frequency distributions of test statistics for synthetic, in-knowledge (IK), and out-of-knowledge (OoK) queries across PubMed and Textbooks corpora.

Limitations and Future Work

While effective, the current framework has limitations. Synthetic queries generated from single document chunks may be simpler than real-world queries requiring multi-document reasoning. Future work will explore "context-aware" chunking and the integration of faithfulness metrics to ensure that LLMs use retrieved information correctly.

Conclusion

This statistical framework provides a robust, training-free method for enhancing RAG reliability. By accurately identifying the boundaries of a system's knowledge, it prevents hallucinations at the source and provides a clear signal for when a knowledge base requires maintenance.

Summary updated

Abstract

Language models (LMs) are known to suffer from hallucinations and misinformation. Retrieval augmented generation (RAG) that retrieves verifiable information from an external knowledge corpus to complement the parametric knowledge in LMs provides a tangible solution to these problems. However, the generation quality of RAG is highly dependent on the relevance between a user's query and the retrieved documents. Inaccurate responses may be generated when the query is outside of the scope of knowledge represented in the external knowledge corpus or if the information in the corpus is out-of-date. In this work, we establish a statistical framework that assesses how well a query can be answered by an RAG system by capturing the relevance of knowledge. We introduce an online testing procedure that employs goodness-of-fit (GoF) tests to inspect the relevance of each user query to detect out-of-knowledge queries with low knowledge relevance. Additionally, we develop an offline testing framework that examines a collection of user queries, aiming to detect significant shifts in the query distribution which indicates the knowledge corpus is no longer sufficiently capable of supporting the interests of the users. We demonstrate the capabilities of these strategies through a systematic evaluation on eight question-answering (QA) datasets, the results of which indicate that the new testing framework is an efficient solution to enhance the reliability of existing RAG systems.
BibTeX
@inproceedings{li2024do,
  author = {Zhuohang Li and Jiaxin Zhang and Chao Yan and Kamalika Das and Kumar Sricharan and Murat Kantarcioglu and Bradley A. Malin},
  title = {{Do You Know What You Are Talking About? Characterizing Query-Knowledge Relevance For Reliable Retrieval Augmented Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.353},
}
Figure