Abstract
Standard pre-trained language models (PLMs) like BERT and GPT excel at capturing linguistic patterns but often fail in "knowledge-intensive" scenarios requiring factual accuracy or common sense. This paper surveys the landscape of Knowledge-Enhanced Models (PLMKEs), categorizing how external knowledge is integrated into neural architectures to mitigate issues like hallucination and static worldviews.
Introduction
While models like BERT and GPT are powerful, they are essentially "closed-book" learners. Their reliance on implicit knowledge absorbed during pre-training leads to significant limitations:
- Static Knowledge: Inability to update information without expensive retraining.
- Hallucination: Generation of factually incorrect content.
- Lack of Common Sense: Difficulty with everyday reasoning not explicitly stated in training corpora.
To address these, researchers have developed PLMKEs that consult external encyclopedias or commonsense knowledge bases, moving toward an "open-book" NLP paradigm.
Taxonomy of Knowledge-Enhanced Models
The integration of knowledge into PLMs can be categorized by three primary dimensions: sources, fusion methods, and reasoning modules.
Knowledge Sources
Knowledge is typically divided into two types: Encyclopedic (factual data) and Commonsense (everyday logic).
Knowledge Fusion Methods
The "how" and "when" of knowledge integration defines the model's architecture:
- Pre-fusion: Knowledge is injected during the initial pre-training phase.
- Post-fusion: The model retrieves relevant knowledge at inference time to help solve a specific task (e.g., RAG or REALM).
- Hybrid-fusion: A combination where knowledge is integrated during both pre-training and inference.
Benchmarks and Tasks
The effectiveness of PLMKEs is evaluated across several specialized tasks:
- Encyclopedic Tasks: Open-domain QA (Natural Questions), Fact Verification (FEVER), and Entity Linking (AIDA CoNLL-YAGO).
- Commonsense Tasks: General reasoning (CommonsenseQA), Social interactions (SocialIQA), and Physical properties (PIQA).
Key Insights
- Fusion Strategy vs. Knowledge Type: Post-fusion (retrieval) is dominant for encyclopedic facts due to the efficiency of looking up specific data. Conversely, pre-fusion is preferred for commonsense reasoning, as it helps the model develop an internal "gut feeling" for implicit logic.
- The Scale Gap: Encyclopedic bases are massive and easily scraped, whereas commonsense sources are smaller and often require manual annotation.
- State-of-the-Art Performance: Hybrid models like RAG and REALM lead in QA, while graph-based models like GEAR excel in fact verification. Large-scale pre-fusion models like ERNIE 3.0 demonstrate broad capabilities across diverse benchmarks.
Future Directions
Despite progress, several challenges remain. Reliability is a major concern, as models may amplify biases present in external sources. Furthermore, most models remain task-specific, highlighting a need for generalist PLMKEs. Finally, distinguishing between true logical reasoning and statistical pattern matching remains a significant hurdle for the field.
Summary updated