Abstract
Pre-trained language models learn informative word representations on a large-scale text corpus through self-supervised learning, which has achieved promising performance in fields of natural language processing (NLP) after fine-tuning. These models, however, suffer from poor robustness and lack of interpretability. We refer to pre-trained language models with knowledge injection as knowledge-enhanced pre-trained language models (KEPLMs). These models demonstrate deep understanding and logical reasoning and introduce interpretability. In this survey, we provide a comprehensive overview of KEPLMs in NLP. We first discuss the advancements in pre-trained language models and knowledge representation learning. Then we systematically categorize existing KEPLMs from three different perspectives. Finally, we outline some potential directions of KEPLMs for future research.
1. Introduction
While Pre-trained Language Models (PLMs) such as BERT and GPT have fundamentally transformed the landscape of Natural Language Processing (NLP), they are not without significant limitations. Often characterized as "stochastic parrots," these models excel at capturing statistical correlations from vast text corpora but frequently struggle with rare vocabulary, complex logical reasoning, and interpretability.
Specifically, standard PLMs face three critical hurdles:
- The Long-Tail Problem: Performance degrades significantly on rare words or entities that appear infrequently in the training data.
- Lack of Logical Reasoning: As statistical models, they often fail at multi-hop reasoning or adhering to explicit symbolic rules.
- The Black Box Problem: It remains difficult to interpret the rationale behind a model's prediction or to update its internal knowledge without prohibitive retraining costs.
To address these challenges, researchers have turned to Knowledge-Enhanced Pre-trained Language Models (KEPLMs). By integrating structured symbolic knowledge—such as Knowledge Graphs (KGs) and logic rules—into neural architectures, we can develop more robust, "world-aware" AI systems.
2. Taxonomy of KEPLMs
The survey provides a systematic taxonomy of KEPLMs categorized across three primary dimensions: granularity, injection method, and parameterization.
2.1 Granularity of Knowledge
This dimension defines the type of knowledge integrated into the model:
- Entity-fused: Focuses on specific objects or concepts.
- Syntax-tree-fused: Incorporates linguistic structures.
- KG-fused: Integrates relational facts from Knowledge Graphs.
- Rule-fused: Embeds explicit logical constraints.
2.2 Method of Knowledge Injection
How knowledge is introduced into the neural architecture varies:
- Feature-fused: Knowledge features are projected directly into input embeddings.
- Embedding-combined: Specialized algorithms (e.g., TransE) create entity embeddings that are fused with word representations.
- Data-structure-unified: Structured triples (Subject-Predicate-Object) are converted into text-like sequences or graphs for native processing.
- Retrieval-based: The model queries an external database during inference (e.g., Retrieval-Augmented Generation).
2.3 Degree of Parameterization
This describes where the knowledge resides:
- Fully Parameterized: Knowledge is baked into the model's weights.
- Partial/Unchanged: Knowledge is kept in its symbolic form to guide the model, preserving its original structure.
3. Knowledge Sources
KEPLMs leverage a variety of external knowledge sources to ground their representations:
- Linguistic/Lexical: Resources like WordNet (synonyms/hierarchies) and SentiWordNet (sentiment polarity).
- Encyclopedic/Factual: Large-scale databases such as Wikipedia, Wikidata, DBpedia, and CN-DBpedia.
- Commonsense: Knowledge bases like ConceptNet (general world knowledge) and ATOMIC (causal reasoning).
- Domain-Specific: Specialized KGs, particularly in the medical field (e.g., BRET-MK).
4. Key Insights and Mechanisms
Integrating symbolic knowledge into neural models presents unique challenges, most notably the "Knowledge-Text Gap"—the difficulty of aligning vector-based text representations with symbolic Knowledge Graph nodes. Models like CoLAKE address this by treating text and KG nodes as a unified graph structure.
Furthermore, knowledge can serve as a Training Signal. Models like KEPLER use knowledge as a supervisor, requiring the model to detect entity replacements, thereby forcing it to learn underlying relations. To prevent "catastrophic forgetting"—where a model loses general language capabilities while learning new facts—researchers utilize Adapters. These are small, modular neural layers that store specific knowledge while keeping the primary model weights frozen.
5. Performance and Evaluation
KEPLMs consistently outperform vanilla PLMs across knowledge-intensive tasks:
- Relation Extraction & Entity Typing: Models such as KnowBERT and ERNIE show significant gains by explicitly recognizing entity boundaries and relationships.
- Question Answering: K-BERT achieved a 1-2% F1 gain in specific domains by integrating relevant KG triples into the input sentence tree.
- Robustness: KEPLMs are generally more resilient to biased prompts and perform better on "long-tail" entities.
6. Future Directions
The survey identifies several frontiers for future KEPLM research:
- Beyond Facts: Moving toward procedural knowledge (how-to) and metacognitive knowledge (self-awareness of certainty).
- Knowledge Updating: Developing methods to update changing facts (e.g., political shifts) without full retraining.
- Conflict Resolution: Handling contradictory information from multiple knowledge sources.
- Abductive Reasoning: Improving the model's ability to perform "inference to the best explanation," a key component of human intelligence.
Summary updated