Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphs

Deepak Nathani, Jatin Chauhan, Charu Sharma, Manohar Kaul

Introduction

Knowledge Graphs (KGs) serve as the backbone for numerous downstream applications, including recommendation systems and question answering. However, even the most comprehensive KGs, such as Freebase and WordNet, are inherently incomplete. Relation prediction—the task of inferring missing links between entities—is therefore a critical area of research. This paper introduces a novel graph attention-based embedding method that significantly improves performance by capturing multi-hop neighborhood features.

The Knowledge Graph Completion Problem

Existing approaches to KG completion typically fall into two categories: translational models and convolutional neural network (CNN) based models. Translational models like TransE are computationally efficient but often fail to capture complex, non-linear relationships. Conversely, CNN-based models like ConvE offer greater expressivity but treat individual triples in isolation, ignoring the rich structural context provided by an entity's local neighborhood.

A Knowledge Graph snippet showing entities, relations, and potential missing links.

As shown in the figure above, an entity's identity is defined by its connections. For instance, knowing that Christopher Nolan directed The Revenant and is a colleague of Tom Hardy provides latent information that can help predict other relations, such as nationality or birthplace. The gap in current research is the failure to account for this local neighborhood context.

Proposed Methodology

The authors propose an encoder-decoder architecture designed to leverage this neighborhood information through a generalized graph attention mechanism.

The Encoder: Generalized Graph Attention

The core of the model is a generalized Graph Attention Network (GAT). Unlike standard GATs that focus solely on node features, this model treats relations as primary features. It represents a triple $(h, r, t)$ by concatenating the embeddings of the head entity, the relation, and the tail entity. A multi-head attention mechanism is then employed to assign weights to neighboring triples, allowing the model to focus on the most relevant contextual information.

To capture long-range dependencies, the model incorporates auxiliary edges between $n$-hop neighbors. These edges represent composed relations, enabling the model to gather information from beyond the immediate neighborhood.

The proposed encoder-decoder architecture, featuring multi-head graph attention and a convolutional decoder.

The Decoder: ConvKB

The context-aware embeddings produced by the encoder are passed to a convolutional decoder (ConvKB). This component analyzes the global properties of the embeddings to produce a final score for a given triple's validity, effectively acting as a classifier for link prediction.

Experimental Results

The model was evaluated against five benchmark datasets: WN18RR, FB15k-237, NELL-995, Kinship, and UMLS. The results demonstrate a substantial leap over previous state-of-the-art (SOTA) methods.

On the challenging FB15k-237 dataset, the model achieved a Mean Reciprocal Rank (MRR) of 0.518 and a Hits@1 of 46.0%. This represents a 104% relative improvement in Hits@1 over previous benchmarks. The model also showed strong performance on dense graphs, where the attention mechanism can effectively sift through high volumes of contextual data to find complex patterns.

Discussion and Insights

Visualization of the attention mechanism reveals that the model's focus evolves during training. Initially, attention is distributed randomly across neighbors. Over time, the model learns to prioritize direct neighbors before eventually expanding its scope to include multi-hop paths and similar relation clusters.

While the model excels in dense environments, it faces limitations in highly sparse or hierarchical structures like WN18RR, where neighborhood-based attention may struggle to capture top-down recursive patterns. Additionally, the inclusion of auxiliary edges for multi-hop paths introduces computational overhead that may impact scalability on extremely large graphs.

Conclusion

By treating relations as first-class citizens and employing a generalized graph attention mechanism, this work sets a new standard for relation prediction in Knowledge Graphs. The massive performance gains on benchmark datasets suggest that capturing local neighborhood context is essential for accurate knowledge base completion. Future research will explore the application of this architecture to textual entailment graphs and the incorporation of higher-order graph motifs.

Summary updated

Abstract

The recent proliferation of knowledge graphs (KGs) coupled with incomplete or partial information, in the form of missing relations (links) between entities, has fueled a lot of research on knowledge base completion (also known as relation prediction). Several recent works suggest that convolutional neural network (CNN) based models generate richer and more expressive feature embeddings and hence also perform well on relation prediction. However, we observe that these KG embeddings treat triples independently and thus fail to cover the complex and hidden information that is inherently implicit in the local neighborhood surrounding a triple. To this effect, our paper proposes a novel attention based feature embedding that captures both entity and relation features in any given entity's neighborhood. Additionally, we also encapsulate relation clusters and multihop relations in our model. Our empirical study offers insights into the efficacy of our attention based model and we show marked performance gains in comparison to state of the art methods on all datasets.
BibTeX
@inproceedings{nathani2019learning,
  author = {Deepak Nathani and Jatin Chauhan and Charu Sharma and Manohar Kaul},
  title = {{Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphs}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=fz4UO7s6xQ},
}
Figure