A Survey of Knowledge-Intensive NLP with Pre-Trained Language Models

Da Yin, Li Dong, Hao Cheng, Xiaodong Liu, Kai-Wei Chang, Furu Wei, Jianfeng Gao

Abstract

Standard pre-trained language models (PLMs) like BERT and GPT excel at capturing linguistic patterns but often fail in "knowledge-intensive" scenarios requiring factual accuracy or common sense. This paper surveys the landscape of Knowledge-Enhanced Models (PLMKEs), categorizing how external knowledge is integrated into neural architectures to mitigate issues like hallucination and static worldviews.

Introduction

While models like BERT and GPT are powerful, they are essentially "closed-book" learners. Their reliance on implicit knowledge absorbed during pre-training leads to significant limitations:

  1. Static Knowledge: Inability to update information without expensive retraining.
  2. Hallucination: Generation of factually incorrect content.
  3. Lack of Common Sense: Difficulty with everyday reasoning not explicitly stated in training corpora.

To address these, researchers have developed PLMKEs that consult external encyclopedias or commonsense knowledge bases, moving toward an "open-book" NLP paradigm.

Taxonomy of Knowledge-Enhanced Models

The integration of knowledge into PLMs can be categorized by three primary dimensions: sources, fusion methods, and reasoning modules.

Knowledge Sources

Knowledge is typically divided into two types: Encyclopedic (factual data) and Commonsense (everyday logic).

Table 1: Categorization of knowledge types, sources, and domains.

Knowledge Fusion Methods

The "how" and "when" of knowledge integration defines the model's architecture:

  • Pre-fusion: Knowledge is injected during the initial pre-training phase.
  • Post-fusion: The model retrieves relevant knowledge at inference time to help solve a specific task (e.g., RAG or REALM).
  • Hybrid-fusion: A combination where knowledge is integrated during both pre-training and inference.
Figure 1: A general framework for knowledge fusion in PLMs, illustrating integration during both pre-training and fine-tuning stages.

Benchmarks and Tasks

The effectiveness of PLMKEs is evaluated across several specialized tasks:

  • Encyclopedic Tasks: Open-domain QA (Natural Questions), Fact Verification (FEVER), and Entity Linking (AIDA CoNLL-YAGO).
  • Commonsense Tasks: General reasoning (CommonsenseQA), Social interactions (SocialIQA), and Physical properties (PIQA).

Key Insights

  1. Fusion Strategy vs. Knowledge Type: Post-fusion (retrieval) is dominant for encyclopedic facts due to the efficiency of looking up specific data. Conversely, pre-fusion is preferred for commonsense reasoning, as it helps the model develop an internal "gut feeling" for implicit logic.
  2. The Scale Gap: Encyclopedic bases are massive and easily scraped, whereas commonsense sources are smaller and often require manual annotation.
  3. State-of-the-Art Performance: Hybrid models like RAG and REALM lead in QA, while graph-based models like GEAR excel in fact verification. Large-scale pre-fusion models like ERNIE 3.0 demonstrate broad capabilities across diverse benchmarks.

Future Directions

Despite progress, several challenges remain. Reliability is a major concern, as models may amplify biases present in external sources. Furthermore, most models remain task-specific, highlighting a need for generalist PLMKEs. Finally, distinguishing between true logical reasoning and statistical pattern matching remains a significant hurdle for the field.

Summary updated

BibTeX
@misc{yin2022survey,
  author = {Da Yin and Li Dong and Hao Cheng and Xiaodong Liu and Kai-Wei Chang and Furu Wei and Jianfeng Gao},
  title = {{A Survey of Knowledge-Intensive NLP with Pre-Trained Language Models}},
  year = {2022},
  eprint = {2202.08772},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2202.08772},
}
Figure