Abstract
Knowledge Graphs (KGs) serve as critical infrastructure for modern AI applications but suffer from inherent incompleteness. While transductive methods are limited to known entities and inductive methods often struggle with domain generalization, this work proposes the inductive KG representation model (iHT). By treating KG completion as a transferable pre-training task on the massive Wikidata5M dataset, iHT achieves a 25% relative improvement over existing state-of-the-art models. The model demonstrates significant zero-shot and few-shot transfer capabilities to unseen graphs, establishing a new foundation for knowledge representation.
1. Introduction
Knowledge Graphs like Wikidata and Freebase are foundational for search and reasoning tasks. However, their utility is hampered by missing facts. Traditional completion methods are often transductive, meaning they learn fixed embeddings for a closed set of entities. This approach fails when new entities are introduced. While inductive methods use textual descriptions to handle unseen entities, they have historically lacked the ability to generalize across different domains. This paper addresses the "generalization gap" by introducing iHT, a model designed to learn the underlying logic of relationships from large-scale data and apply it to novel contexts.
2. Methodology: The iHT Architecture
The core of the proposed approach is the inductive KG representation model (iHT), which utilizes a dual-Transformer architecture to bridge the gap between language and graph structure.
2.1 Dual-Transformer Design
The model consists of two primary components:
- Entity Transformer: This module encodes the "surface form" (textual descriptions or names) of an entity into a dense vector. It is initialized with pre-trained BERT weights, leveraging deep linguistic knowledge to understand entity semantics.
- Context Transformer: This module serves as the reasoning engine. It integrates the entity embedding with a specific relational query and the entity’s local graph neighborhood.
2.2 Training Strategy
Unlike simpler scoring functions such as TransE, iHT employs the full expressive power of Transformers to model multi-relational interactions. The model is trained using in-batch negative sampling, allowing for efficient contrastive learning against thousands of alternative facts simultaneously.
3. Experimental Evaluation
The researchers evaluated iHT across three diverse datasets:
- Wikidata5M: A large-scale KG with 5 million entities and 20 million triplets, used for primary pre-training.
- FB15K-237: A standard benchmark derived from Freebase for testing cross-distribution transfer.
- WN18RR: A lexical relation dataset based on WordNet, used to test domain-specific generalization.
4. Key Results and Insights
The iHT model sets a new benchmark for KG completion, particularly in its ability to function as a "foundation model" for relational data.
4.1 State-of-the-Art Performance
On the Wikidata5M benchmark, iHT outperformed previous leaders—including KGT5 and KEPLER—by over 25% in Mean Reciprocal Rank (MRR). This highlights the effectiveness of combining deep entity encoding with context-aware reasoning.
4.2 Transferability and Efficiency
A key finding is iHT's superior transferability. When fine-tuned on smaller datasets like FB15K-237, the Wikidata-pre-trained model consistently outperformed models trained from scratch. Furthermore, iHT demonstrated high low-resource efficiency: with only 10% of training data, it achieved up to 86% of its full-data performance.
4.3 Architectural Observations
Ablation studies revealed that a deep Entity Transformer (9 layers) combined with a shallow Context Transformer (3 layers) yielded optimal results. This suggests that capturing entity meaning from text is more complex than modeling local graph topology. Additionally, the use of full textual descriptions was found to be critical; models relying solely on entity names suffered significant performance degradation due to ambiguity.
5. Conclusion
iHT represents a significant step toward general-purpose Knowledge Graph models. By leveraging large-scale pre-training and Transformer architectures, it overcomes the limitations of transductive and domain-specific inductive methods. While the model's reliance on textual descriptions and its computational intensity during inference are noted limitations, its success in transfer learning opens new avenues for downstream tasks like multi-hop reasoning and complex question answering.
Summary updated