TL;DR
This paper introduces KGLM, a novel approach that enhances language models for knowledge graph completion by incorporating explicit entity and relation type embeddings during pre-training. This integration allows the model to better understand the inherent structure of knowledge graphs, leading to state-of-the-art performance in link prediction tasks.
What Problem Does This Paper Address?
Knowledge graphs (KGs) are powerful tools for representing complex relationships, but they are often incomplete. Knowledge graph completion (KGC) aims to fill these gaps, with link prediction being a key task. While pre-trained language models (PLMs) like BERT have shown promise in KGC by treating triples as text, they often overlook the crucial structural information within KGs, such as entity and relation types. This oversight limits their ability to fully leverage the KG's inherent structure, especially when dealing with inverse relations or disambiguating entities. This paper addresses this gap by proposing a method to integrate KG structure directly into the PLM pre-training process.
Methods
The core innovation of KGLM lies in its Knowledge Graph Language Model (KGLM) architecture. This approach extends existing PLMs (specifically RoBERTa in their experiments) by introducing a new entity/relation-type embedding layer. This layer learns unique embeddings for different entity and relation types, allowing the model to differentiate them.
The process involves:
- Pre-training Corpus Generation: Both forward and inverse triples are extracted from the KG to create a pre-training corpus.
- Continued Pre-training: The PLM is further pre-trained using a masked language model (MLM) objective on this KG-derived corpus. Crucially, this pre-training phase incorporates the new entity/relation-type embedding layer alongside the standard token and position embeddings.
- Fine-tuning: After pre-training, the model is fine-tuned on the specific link prediction task. The fine-tuning stage uses a weighted average of scores from both forward and inverse triple directions to leverage the learned structural information.
The authors explore three variations of the ER-type embeddings: KGLMBase (simplified entity/relation types), KGLMGR (granular relation types), and KGLMGER (most granular, considering all entity and relation types).
Datasets
The paper evaluates KGLM on three benchmark datasets for link prediction:
- WN18RR: Derived from WordNet, focusing on semantic relationships.
- FB15k-237: Extracted from Freebase, containing general facts about the world.
- UMLS: Contains biomedical relationships.
These datasets are subsets of WN18 and FB15k, respectively, with corrections made to address inverse relation test leakage.
Key Insights and Ideas
- Structure Matters: Explicitly encoding entity and relation types during PLM pre-training significantly improves performance on link prediction tasks, demonstrating that language models can benefit from structured KG information beyond just textual representations.
- Pre-training is Key: Integrating the ER-type embeddings during the pre-training phase, rather than just fine-tuning, yields superior results. This suggests that learning structural nuances early on is more effective.
- Inverse Relations are Important: The KGLM architecture's ability to handle both forward and inverse triples (e.g.,
daughterOfanddaughterOf⁻¹) is crucial for capturing the full spectrum of relationships in a KG. - Synergy of Methods: Combining the strengths of latent feature models (like TransE) and textual encoding models (like BERT) through structural embedding integration leads to a more robust and performant model.
Key Results
KGLM achieved state-of-the-art results on the WN18RR dataset for link prediction. Specifically:
- WN18RR: KGLMGR improved the mean rank (MR) by 21.2% compared to the previous state-of-the-art (40.18 vs. 51). It also showed a 35.8% improvement in hits@1 compared to the best language model-based approach (StAR).
- UMLS: KGLM achieved the best performance across all metrics, with a 21.2% improved MR compared to ComplEx (1.19 vs. 1.51).
- FB15k-237: While not showing improvement over StAR on this dataset, KGLM still performed competitively.
- Ablation Studies: Table 3 demonstrates that the combination of continued pre-training with ER-type embeddings and fine-tuning is essential for maximizing performance, outperforming models that only use one of these components.
Limitations and Future Work
The authors acknowledge a significant limitation: the long training and inference time associated with their approach. They plan to address this in future work by exploring Siamese-style textual encoders.
Future work also includes testing the more granular KGLMGER version on domain-specific knowledge graphs that explicitly define entity types, such as KIDS, to further explore the benefits of type-aware embeddings.
Summary updated