Pre-training Transformers for Knowledge Graph Completion

Sanxing Chen, Hao Cheng, Xiaodong Liu, Jian Jiao, Yangfeng Ji, Jianfeng GaoarXiv 2023 · DOI · arXiv

Abstract

Knowledge Graphs (KGs) serve as critical infrastructure for modern AI applications but suffer from inherent incompleteness. While transductive methods are limited to known entities and inductive methods often struggle with domain generalization, this work proposes the inductive KG representation model (iHT). By treating KG completion as a transferable pre-training task on the massive Wikidata5M dataset, iHT achieves a 25% relative improvement over existing state-of-the-art models. The model demonstrates significant zero-shot and few-shot transfer capabilities to unseen graphs, establishing a new foundation for knowledge representation.

1. Introduction

Knowledge Graphs like Wikidata and Freebase are foundational for search and reasoning tasks. However, their utility is hampered by missing facts. Traditional completion methods are often transductive, meaning they learn fixed embeddings for a closed set of entities. This approach fails when new entities are introduced. While inductive methods use textual descriptions to handle unseen entities, they have historically lacked the ability to generalize across different domains. This paper addresses the "generalization gap" by introducing iHT, a model designed to learn the underlying logic of relationships from large-scale data and apply it to novel contexts.

2. Methodology: The iHT Architecture

The core of the proposed approach is the inductive KG representation model (iHT), which utilizes a dual-Transformer architecture to bridge the gap between language and graph structure.

Conceptual overview of transductive vs. inductive KG completion and the iHT model architecture.

2.1 Dual-Transformer Design

The model consists of two primary components:

  1. Entity Transformer: This module encodes the "surface form" (textual descriptions or names) of an entity into a dense vector. It is initialized with pre-trained BERT weights, leveraging deep linguistic knowledge to understand entity semantics.
  2. Context Transformer: This module serves as the reasoning engine. It integrates the entity embedding with a specific relational query and the entity’s local graph neighborhood.

2.2 Training Strategy

Unlike simpler scoring functions such as TransE, iHT employs the full expressive power of Transformers to model multi-relational interactions. The model is trained using in-batch negative sampling, allowing for efficient contrastive learning against thousands of alternative facts simultaneously.

3. Experimental Evaluation

The researchers evaluated iHT across three diverse datasets:

  • Wikidata5M: A large-scale KG with 5 million entities and 20 million triplets, used for primary pre-training.
  • FB15K-237: A standard benchmark derived from Freebase for testing cross-distribution transfer.
  • WN18RR: A lexical relation dataset based on WordNet, used to test domain-specific generalization.

4. Key Results and Insights

The iHT model sets a new benchmark for KG completion, particularly in its ability to function as a "foundation model" for relational data.

Performance comparison of iHT against baseline models on the Wikidata5M dataset.

4.1 State-of-the-Art Performance

On the Wikidata5M benchmark, iHT outperformed previous leaders—including KGT5 and KEPLER—by over 25% in Mean Reciprocal Rank (MRR). This highlights the effectiveness of combining deep entity encoding with context-aware reasoning.

4.2 Transferability and Efficiency

A key finding is iHT's superior transferability. When fine-tuned on smaller datasets like FB15K-237, the Wikidata-pre-trained model consistently outperformed models trained from scratch. Furthermore, iHT demonstrated high low-resource efficiency: with only 10% of training data, it achieved up to 86% of its full-data performance.

4.3 Architectural Observations

Ablation studies revealed that a deep Entity Transformer (9 layers) combined with a shallow Context Transformer (3 layers) yielded optimal results. This suggests that capturing entity meaning from text is more complex than modeling local graph topology. Additionally, the use of full textual descriptions was found to be critical; models relying solely on entity names suffered significant performance degradation due to ambiguity.

5. Conclusion

iHT represents a significant step toward general-purpose Knowledge Graph models. By leveraging large-scale pre-training and Transformer architectures, it overcomes the limitations of transductive and domain-specific inductive methods. While the model's reliance on textual descriptions and its computational intensity during inference are noted limitations, its success in transfer learning opens new avenues for downstream tasks like multi-hop reasoning and complex question answering.

Summary updated

Abstract

Learning transferable representation of knowledge graphs (KGs) is challenging due to the heterogeneous, multi-relational nature of graph structures. Inspired by Transformer-based pretrained language models' success on learning transferable representation for texts, we introduce a novel inductive KG representation model (iHT) for KG completion by large-scale pre-training. iHT consists of a entity encoder (e.g., BERT) and a neighbor-aware relational scoring function both parameterized by Transformers. We first pre-train iHT on a large KG dataset, Wikidata5M. Our approach achieves new state-of-the-art results on matched evaluations, with a relative improvement of more than 25% in mean reciprocal rank over previous SOTA models. When further fine-tuned on smaller KGs with either entity and relational shifts, pre-trained iHT representations are shown to be transferable, significantly improving the performance on FB15K-237 and WN18RR.
BibTeX
@misc{chen2023pre,
  author = {Sanxing Chen and Hao Cheng and Xiaodong Liu and Jian Jiao and Yangfeng Ji and Jianfeng Gao},
  title = {{Pre-training Transformers for Knowledge Graph Completion}},
  year = {2023},
  eprint = {2303.15682},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.15682},
}
Figure