Abstract
The recent proliferation of knowledge graphs (KGs) coupled with incomplete or partial information, in the form of missing relations (links) between entities, has fueled a lot of research on knowledge base completion (also known as relation prediction). Several recent works suggest that convolutional neural network (CNN) based models generate richer and more expressive feature embeddings and hence also perform well on relation prediction. However, we observe that these KG embeddings treat triples independently and thus fail to cover the complex and hidden information that is inherently implicit in the local neighborhood surrounding a triple. To this effect, our paper proposes a novel attention based feature embedding that captures both entity and relation features in any given entity's neighborhood. Additionally, we also encapsulate relation clusters and multihop relations in our model. Our empirical study offers insights into the efficacy of our attention based model and we show marked performance gains in comparison to state of the art methods on all datasets.
Introduction
Knowledge Graphs (KGs) like Freebase or WordNet serve as critical infrastructure for semantic search and question-answering. Despite their utility, they are often incomplete. Relation prediction aims to infer missing links between entities. Existing models generally fall into two categories: translational models (e.g., TransE) and CNN-based models (e.g., ConvE). While CNN-based models are expressive, they are often "myopic," treating each triple $(head, relation, tail)$ as an independent event.
The core limitation of these approaches is the neglect of the local neighborhood context. In a real-world KG, the fact that Christopher Nolan directed Inception is contextually linked to Inception starring Leonardo DiCaprio. By failing to encapsulate these semantically rich local relations, current models miss the structural "vibe" of the graph necessary for accurate link prediction.
Methodology
The proposed architecture utilizes an encoder-decoder framework that treats the KG as a graph structure rather than a flat list of facts.
1. The Encoder: Generalized Graph Attention Network (GAT)
Unlike standard GATs that focus solely on node (entity) features, this model incorporates relation features directly into the attention mechanism. It calculates an attention score for every triple in an entity's neighborhood, allowing the model to dynamically weight the most relevant neighbors during embedding generation.
2. Multi-hop Reasoning
To mitigate the "vanishing gradient" problem common in deep graph networks, the authors introduce auxiliary edges. These edges connect $n$-hop neighbors directly, facilitating efficient information flow across the graph and capturing long-range dependencies.
3. The Decoder: ConvKB
The context-aware embeddings produced by the encoder are passed to a convolutional decoder (ConvKB). This component analyzes global embedding properties across dimensions to compute a final score for the validity of a predicted triple.
Experiments and Results
The model was evaluated on five benchmark datasets, including WN18RR, FB15k-237, and NELL-995. The results demonstrate that incorporating neighborhood context leads to significant performance improvements:
- FB15k-237: The model achieved a Hits@1 of 46.0%, more than doubling the performance of the previous state-of-the-art, ConvE (22.5%).
- NELL-995: The model outperformed all baselines across MRR, Hits@1, @3, and @10.
- Ablation Studies: Removing relation information or multi-hop auxiliary edges resulted in significant performance drops, validating the importance of the neighborhood-centric approach.
Key Insights
- Relations as First-Class Citizens: By concatenating relation embeddings with entity embeddings during the attention phase, the model learns the specific nature of connections, not just their existence.
- Dynamic Role Representation: An entity's representation evolves based on its neighbors, capturing the diverse roles an entity might play (e.g., a person as both a "director" and a "colleague").
- Attention Evolution: Visualization of attention weights over training epochs reveals that the model learns to filter noise and focus on semantically relevant multi-hop paths as it converges.
Conclusion and Future Work
While the model excels in dense graphs like FB15k-237, it faces challenges in sparse or strictly hierarchical structures like WN18RR. Future research will focus on better handling hierarchical graphs, capturing higher-order graph motifs, and applying these embeddings to downstream tasks such as dialogue generation and complex question answering.
Summary updated