Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applications

Zhangyin Feng, Weitao Ma, Weijiang Yu, Lei Huang, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, Ting Liu

TL;DR

Large Language Models (LLMs) excel at many tasks but struggle with outdated or incomplete knowledge. This paper surveys the two main approaches to address this: knowledge editing (modifying LLM parameters) and retrieval augmentation (fetching external information). It provides a taxonomy of methods, benchmarks, and applications, offering a comprehensive overview to guide future research.

What Problem Does This Paper Address?

LLMs, despite their impressive capabilities, have inherent limitations. They are trained on static datasets, meaning their knowledge becomes outdated, and they struggle with "long-tail" knowledge (less common facts). Furthermore, they can "hallucinate" or generate incorrect information. These issues are particularly problematic for knowledge-intensive tasks. To overcome these limitations, researchers are exploring ways to inject external, up-to-date knowledge into LLMs. This paper addresses the need for a structured overview of the diverse and rapidly evolving landscape of methods that integrate external knowledge with LLMs.

Methods

The paper categorizes the integration of knowledge and LLMs into two primary strategies:

  • Knowledge Editing: This approach aims to directly modify the LLM's internal knowledge. It's further broken down into:
  • Input Editing: Modifying the input prompt to include external information or refining the prompt based on feedback. This is cost-effective and preserves the original model.
  • Model Editing: Directly altering the LLM's parameters. This includes:
  • Knowledge Plug-in: Adding external parametric knowledge without altering original weights (e.g., using adapter layers).
  • Locate-then-Edit: Identifying specific parameters responsible for certain knowledge and modifying them.
  • Overall Editing: Directly modifying the model's parameters without locating specific knowledge storage.
  • Assess Knowledge Editing: Evaluating the effectiveness of editing through metrics like reliability, generality, and locality.
  • Retrieval Augmentation: This strategy keeps the LLM's parameters unchanged and fetches relevant external information during inference. Key components include:
  • Retrieval Judgement: Determining when an LLM needs external knowledge.
  • Document Retrieval: Fetching relevant documents from external sources (e.g., Wikipedia) using retrievers or generating documents with LLMs.
  • Document Utilization: How retrieved documents are used, categorized into:
  • Input Enhancement: Prepending documents to the input prompt.
  • Reasoning Verification: Using documents to check the LLM's reasoning process.
  • Answer Verification: Modifying the LLM's final answer based on retrieved evidence.
  • Knowledge Conflict Resolution: Addressing inconsistencies between the LLM's internal knowledge and retrieved documents, or between different retrieved documents.

Datasets

The paper discusses various benchmarks used for evaluating knowledge editing and retrieval augmentation. For knowledge editing, notable datasets include:

  • ZSRE, CounterFact, CounterFact+: Used for evaluating reliability, generality, and locality.
  • Bi-ZSRE: Used for cross-lingual evaluation.
  • MQUAKE, RippleEdits, Eva-KELLM: Benchmarks for assessing specific aspects of knowledge editing like ripple effects and cross-lingual capabilities.

For retrieval augmentation, benchmarks are categorized by task type:

  • Single-hop QA: Natural Questions (NQ), TriviaQA, PopQA.
  • Multi-hop QA: HotPotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle.
  • Fact Verification: Fever, Feverous, FoolMeTwice.
  • Complex Reasoning: StrategyQA, CommonsenseQA, CommonsenseQA2.0, CSQA, TempQuestions, INFOTABS.

Key Insights and Ideas

  • LLMs are not static knowledge bases: Their knowledge is fixed at training time, leading to outdated information and a struggle with rare facts.
  • Two main paths to knowledge integration: Knowledge editing directly modifies the LLM, while retrieval augmentation fetches external data on the fly. Both have distinct advantages and disadvantages.
  • Input editing is a low-cost entry point: Modifying prompts is simple and effective for injecting information without retraining.
  • Model editing offers persistence: Directly changing model parameters ensures knowledge is integrated more deeply, but is more complex.
  • Retrieval augmentation is flexible: It allows LLMs to access up-to-date information without altering their core. However, it introduces challenges in deciding when to retrieve and how to handle conflicting information.
  • Evaluating knowledge integration is complex: Metrics like reliability, generality, and locality are crucial, and cross-lingual evaluation is an emerging challenge.
  • Knowledge conflicts are a significant hurdle: Inconsistencies between internal and external knowledge, or among external sources, need robust solutions.

Key Results

The paper is a survey and taxonomy, so it doesn't present new quantitative results. Instead, it synthesizes findings from numerous prior works. The key "results" are the organization and categorization of the field, highlighting:

  • A comprehensive taxonomy of knowledge editing and retrieval augmentation methods.
  • A detailed overview of existing benchmarks for evaluating these methods.
  • Identification of common applications like New Bing.
  • An analysis of the strengths and weaknesses of different approaches.

Limitations and Future Work

The authors acknowledge that the field is still in its early stages and leaves much room for improvement. Key future directions include:

  • Multi-source Knowledge Augmentation: Developing methods to integrate knowledge from diverse sources and formats, not just triples or unstructured text.
  • Knowledge Augmented Multi-modal LLMs: Extending LLMs to reason with and integrate knowledge from multiple modalities (text, images, etc.).
  • LLM-Based Agents: Enhancing autonomous agents with knowledge integration capabilities for complex real-world decision-making.
  • Analysis of Knowledge Enhancement Methods: Moving beyond evaluating generation results to a more comprehensive analysis of side effects and the viability of different methods compared to retrieval-based architectures.
  • Resolving Knowledge Conflicts: Developing more robust methods for filtering data, fine-tuning models, and handling inconsistencies.

Summary updated

Abstract

Large language models (LLMs) exhibit superior performance on various natural language tasks, but they are susceptible to issues stemming from outdated data and domain-specific limitations. In order to address these challenges, researchers have pursued two primary strategies, knowledge editing and retrieval augmentation, to enhance LLMs by incorporating external information from different aspects. Nevertheless, there is still a notable absence of a comprehensive survey. In this paper, we propose a review to discuss the trends in integration of knowledge and large language models, including taxonomy of methods, benchmarks, and applications. In addition, we conduct an in-depth analysis of different methods and point out potential research directions in the future. We hope this survey offers the community quick access and a comprehensive overview of this research area, with the intention of inspiring future research endeavors.
BibTeX
@misc{feng2023trends,
  author = {Zhangyin Feng and Weitao Ma and Weijiang Yu and Lei Huang and Haotian Wang and Qianglong Chen and Weihua Peng and Xiaocheng Feng and Bing Qin and Ting Liu},
  title = {{Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applications}},
  year = {2023},
  eprint = {2311.05876},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.05876},
}
Figure