On the Role of Long-tail Knowledge in Retrieval Augmented Large Language Models

Dongyang Li, Junbing Yan, Taolin Zhang, Chengyu Wang, Xiaofeng He, Longtao Huang, Hui Xue, Jun Huang

Abstract

Retrieval augmented generation (RAG) exhibits outstanding performance in promoting the knowledge capabilities of large language models (LLMs) with retrieved documents related to user queries. However, RAG only focuses on improving the response quality of LLMs via enhancing queries indiscriminately with retrieved information, paying little attention to what type of knowledge LLMs really need to answer original queries more accurately. In this paper, we suggest that long-tail knowledge is crucial for RAG as LLMs have already remembered common world knowledge during large-scale pre-training. Based on our observation, we propose a simple but effective long-tail knowledge detection method for LLMs. Specifically, the novel Generative Expected Calibration Error (GECE) metric is derived to measure the "long-tailness" of knowledge based on both statistics and semantics. Hence, we retrieve relevant documents and infuse them into the model for patching knowledge loopholes only when the input query relates to long-tail knowledge. Experiments show that, compared to existing RAG pipelines, our method achieves over 4x speedup in average inference time and consistent performance improvement in downstream tasks.

Introduction

Retrieval-Augmented Generation (RAG) has become a cornerstone for mitigating hallucinations in Large Language Models (LLMs) by grounding responses in external, verifiable documents. Despite its success, current RAG implementations suffer from a significant inefficiency: they are "indiscriminate." These systems retrieve external information for every query, even when the model already possesses the necessary knowledge within its internal parameters. This leads to unnecessary computational overhead and can introduce noise that degrades the quality of responses for common knowledge queries.

The authors propose that RAG should be targeted specifically at long-tail knowledge—rare or specialized information that the model likely missed or under-represented during its pre-training phase. By identifying these knowledge "loopholes," retrieval can be triggered only when truly necessary.

Methodology: The GECE Metric

To implement selective retrieval, the authors introduce the Generative Expected Calibration Error (GECE). This metric serves as a detector for long-tail knowledge by combining four distinct signals:

  1. Semantic Accuracy: Measured via the METEOR score, assessing the alignment between raw model predictions and ground truth.
  2. Model Confidence: The average probability assigned to generated tokens.
  3. Statistical Frequency: Incorporating word frequency ($\alpha$), where rare words signal long-tail content.
  4. Gradient Disparity: A technical measure of how much an instance's gradient deviates from the dataset's mean gradient, indicating mathematical distinctness in the latent space.
Conceptual overview of selective retrieval based on knowledge type.

The resulting pipeline calculates GECE for each incoming query. High GECE values trigger retrieval from external sources (like Wikipedia), while low values allow the model to answer directly, bypassing the retrieval step to maximize efficiency.

Experimental Evaluation

The method was evaluated on three benchmarks: Natural Questions (NQ), TriviaQA, and MMLU. These datasets provide a range of difficulty from common web searches to professional-level multi-subject understanding.

Results and Discussion

The experiments demonstrate that selective retrieval is not just a trade-off but a "win-win" for both speed and accuracy.

Performance and speed-up metrics across various RAG models with and without GECE.

As shown in the results, the GECE-enhanced pipeline achieved massive speedups, ranging from 2.1x to 7.3x across different base models like Llama2-7B and complex RAG frameworks like ITER-RETGEN. Notably, performance metrics (Rouge-1 and Bleu-4) also improved. For instance, on the NQ dataset, Llama2-7B's Rouge-1 score rose from 42.1 to 42.9 while doubling in speed. This suggests that filtering out redundant retrieval for "head" knowledge reduces noise and allows the model to focus its reasoning capabilities more effectively.

Conclusion

This research highlights that LLMs are often "knowledge-saturated" for common facts, making indiscriminate retrieval redundant. By using GECE to detect long-tail knowledge, RAG systems can achieve significant architectural efficiency. While calculating gradients introduces some overhead, the overall gains in inference speed and response quality make a compelling case for selective, knowledge-aware retrieval systems. Future work will explore the generalizability of GECE across more diverse domains and architectures.

Summary updated

Abstract

Retrieval augmented generation (RAG) exhibits outstanding performance in promoting the knowledge capabilities of large language models (LLMs) with retrieved documents related to user queries. However, RAG only focuses on improving the response quality of LLMs via enhancing queries indiscriminately with retrieved information, paying little attention to what type of knowledge LLMs really need to answer original queries more accurately. In this paper, we suggest that long-tail knowledge is crucial for RAG as LLMs have already remembered common world knowledge during large-scale pre-training. Based on our observation, we propose a simple but effective long-tail knowledge detection method for LLMs. Specifically, the novel Generative Expected Calibration Error (GECE) metric is derived to measure the ``long-tailness'' of knowledge based on both statistics and semantics. Hence, we retrieve relevant documents and infuse them into the model for patching knowledge loopholes only when the input query relates to long-tail knowledge. Experiments show that, compared to existing RAG pipelines, our method achieves over 4x speedup in average inference time and consistent performance improvement in downstream tasks.
BibTeX
@inproceedings{li2024role,
  author = {Dongyang Li and Junbing Yan and Taolin Zhang and Chengyu Wang and Xiaofeng He and Longtao Huang and Hui Xue and Jun Huang},
  title = {{On the Role of Long-tail Knowledge in Retrieval Augmented Large Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-short.12},
}
Figure