TL;DR
This paper introduces TransT, a novel approach for knowledge graph completion (KGC) that leverages entity types to generate multiple, context-aware embeddings for each entity. By integrating type-based semantic similarity, TransT effectively addresses the polysemy of entities and improves the accuracy of predicting missing links in knowledge graphs.
What Problem Does This Paper Address?
Knowledge graphs (KGs) are crucial for AI applications, but they are often incomplete. Knowledge graph completion (KGC) aims to fill these gaps by predicting missing entity-relation triples. Existing methods primarily rely on the structured information within triples, often neglecting the rich semantic information of entities and relations, such as their types and descriptions. This neglect leads to two main problems:
- Entity Polysemy: Entities can have different meanings in different contexts (e.g., "Newton" as a person vs. a writer). Representing each entity with a single vector struggles to capture these diverse semantics.
- Limited Prior Knowledge: Most methods optimize for maximum likelihood, overlooking the prior probability of entities and relations. This can lead to less accurate predictions, especially when inferring missing elements.
TransT aims to overcome these limitations by incorporating entity type information to create more nuanced entity representations and better prior probability estimations.
Methods
TransT introduces a type-based multiple embedding approach for KGC. The core innovations include:
- Type-Based Semantic Similarity: The paper constructs "relation types" by considering the common types of entities that appear in the head and tail positions of a relation. This allows for measuring semantic similarity between entities, relations, and their types using a Jaccard-like index.
- Multiple Embedding Representations: Instead of a single vector, each entity is represented by a set of semantic vectors. This allows TransT to capture the multiple semantics of an entity in different contexts. The number of semantic vectors for each entity is dynamically generated using a Chinese Restaurant Process (CRP) to avoid manual subjectivity.
- Prior Probability Estimation: The type-based semantic similarity is used to estimate prior probabilities for predicting missing elements. The intuition is that entities with more common types related to the given context are more likely to be the missing element.
- Objective Function: TransT optimizes a posterior probability estimation, combining a likelihood term (derived from the multiple embeddings) and a prior probability term (derived from semantic similarity). Negative sampling is employed to train the model efficiently.
Datasets
The proposed TransT model is evaluated on two widely used benchmark datasets for KGC:
- FB15K: A subset of the Freebase knowledge graph. It contains explicit type information for entities.
- WN18: A subset of the WordNet knowledge graph. Explicit type information is not directly available, so the authors construct entity types from lexical categories.
Key Insights and Ideas
- Types as a Rich Source of Information: Entity types, while simpler than full descriptions, provide a powerful and less noisy signal for understanding entity semantics and relationships.
- Multiple Embeddings for Polysemy: Representing entities with multiple vectors is crucial for accurately modeling their diverse meanings across different relational contexts.
- Prior Knowledge Enhances Prediction: Incorporating type-based prior probabilities significantly improves KGC performance by guiding predictions towards more semantically plausible entities or relations.
- Type Information is More Discriminative in Simpler KGs: The benefits of type information are more pronounced in datasets like WN18, where types are simpler and entities have fewer types, leading to better distinction.
Key Results
TransT demonstrates significant improvements over existing state-of-the-art methods on both FB15K and WN18 datasets for entity prediction, relation prediction, and triple classification.
- Entity Prediction: TransT achieved the best performance on FB15K with the "filter" setting (achieving 85.4% HITS@10), significantly outperforming baselines. On WN18, it achieved 92.7% HITS@10 (Raw) and 97.4% HITS@10 (Filter).
- Relation Prediction: TransT achieved a Mean Rank of 1.19 and HITS@1 of 94.1% on FB15K, outperforming TKRL by 3.5% in HITS@1 and 0.88 in Mean Rank.
- Triple Classification: TransT achieved an accuracy of 91.0% on FB15K, surpassing all baselines and being the only model to exceed 90% accuracy.
The paper also shows that the number of semantic vectors for an entity correlates with its complexity (number of relations, types, and triples), suggesting that TransT's approach aligns with intuitive notions of entity semantics.
Limitations and Future Work
- Parameter Sensitivity: The paper mentions that parameters like similarity weights ($\lambda$) may need to be adjusted for different datasets.
- Type Construction for WN18: While effective, the method for constructing types in WN18 is a workaround for the lack of explicit type information.
- Future Work: The authors suggest that their approach can be extended to incorporate other semantic information like descriptions and lexical categories, and to handle more complex KG structures. The dynamic generation of semantic vectors using CRP also opens avenues for further exploration in modeling entity semantics.
Summary updated