Abstract
Knowledge Graphs (KGs) are fundamental to many AI applications but are inherently incomplete and dynamic. Traditional KG embedding models, such as TransE, assign unique vectors to entities during training, making them incapable of representing "Out-of-KG" entities that appear post-training. This paper introduces JointE, a model designed for zero-shot entity prediction. JointE bridges the gap for unseen entities by learning to map their textual descriptions into a shared semantic embedding space. By moving beyond simple translation-based models to a pairwise interaction approach, JointE achieves state-of-the-art performance on zero-shot entity prediction tasks.
Introduction
The evolution of Knowledge Graphs like Freebase and WordNet necessitates models that can handle a constant influx of new information. The primary challenge is the zero-shot learning problem: predicting relationships for entities never encountered during the model's training phase.
To address this, the authors leverage the rich textual descriptions often accompanying KG entities. These descriptions serve as a bridge, allowing the model to generate embeddings for unseen nodes based on their semantic content rather than their position in a pre-defined graph structure.
The JointE Model
JointE jointly learns embeddings for entities, relations, and the constituent words of entity descriptions. The model architecture is built upon several key innovations:
1. Zero-Shot Embedding via WBOW
To represent an unseen entity, JointE utilizes a Weighted Bag of Words (WBOW) approach. It calculates TF-IDF weights for words within an entity's description and aggregates their individual word embeddings into a single entity vector. This method is computationally efficient and has proven more robust than complex neural architectures for short descriptions.
2. Pairwise Interaction Scoring
Unlike traditional "translation-based" models ($subject + predicate \approx object$), JointE employs a scoring function based on three distinct pairwise correlations:
- Correlation between subject and predicate ($f_{sp}$).
- Correlation between predicate and object ($f_{po}$).
- Correlation between subject and object ($f_{so}$).
3. Dual Relation Embeddings
To capture the directed nature of KG facts (e.g., "Ithaca is in New York" vs. "New York is in Ithaca"), JointE represents each relation $p$ with two vectors: $p_s$ for subject interaction and $p_o$ for object interaction.
4. Optimization
The model is trained using a Contrastive Max-Margin (CMM) hinge loss. This objective function ensures that valid triples are scored higher than "corrupted" triples, where one entity is replaced by a random alternative.
Experimental Evaluation
The model was evaluated on two modified benchmarks tailored for zero-shot scenarios:
- FB20K: A Freebase subset with ~20,000 entities and 1,341 relations, including over 5,000 "Out-of-KG" entities for testing.
- WN35K: A WordNet subset with ~35,000 entities and 18 relations.
Testing covered three specific scenarios: (d-e) new subject, (e-d) new object, and (d-d) where both subject and object are unseen.
Results and Discussion
JointE demonstrated significant improvements over previous state-of-the-art methods, such as DKRL. The primary evaluation metric was Hits@10 (the percentage of correct entities in the top 10 predictions).
- FB20K Performance: JointE achieved a Hits@10 of 42.9%, compared to DKRL(CNN)'s 29.5%.
- WN35K Performance: JointE reached 35.9%, outperforming DKRL's 28.6%.
- Zero-Shot Gains: In the most challenging (d-d) scenario on FB20K, JointE's performance jumped from 72.5% to 85.6%.
Key Insights
- Interaction vs. Translation: The dot-product interaction between triple components captures more complex semantic dependencies than restrictive translation models.
- WBOW Robustness: The success of WBOW over CNNs suggests that for short KG descriptions, keyword presence is more critical than syntactic structure.
- The Semantic Gap: A performance discrepancy remains between In-KG entities (learned from graph structure) and Out-of-KG entities (derived from text). Aligning these two distinct embedding sources is a primary area for future research.
Conclusion
JointE provides a powerful framework for incorporating unseen entities into Knowledge Graphs by leveraging textual metadata. While the "semantic gap" between graph-based and text-based embeddings remains a challenge, the model's pairwise interaction scoring sets a new benchmark for zero-shot entity prediction. Future work may involve integrating pre-trained Transformer models to further enhance entity representations from text.
Summary updated