Knowledge Graph Embedding using Graph Convolutional Networks with Relation-Aware Attention

Nasrullah Sheikh, Xiao Qin, Berthold Reinwald, Christoph Miksovic, Thomas Gschwind, Paolo Scotton

Introduction

Knowledge Graphs (KGs) serve as the backbone for many modern AI applications, representing complex real-world facts as triplets of (head entity, relation, tail entity). However, most KGs are inherently incomplete, necessitating robust embedding techniques to predict missing links and match entities across disparate sources. While Graph Convolutional Networks (GCNs) have become a standard tool for learning graph representations, they often fail to account for the semantic diversity of relationships in a graph. This paper introduces RelAtt, a relation-aware attention mechanism designed to make GCNs sensitive to the specific types of connections between entities.

The Challenge of Relation Heterogeneity

In a standard GCN, an entity's representation is updated by aggregating features from all its neighbors, typically treating every edge with equal importance. In complex enterprise knowledge graphs, this approach is suboptimal. For instance, a relationship like subsidiary_of might be far more critical for identifying a company's structure than a generic in_industry link. By ignoring these semantic differences, standard GCNs produce embeddings that lack the nuance required for high-precision tasks. RelAtt addresses this by teaching the network to dynamically weight neighbors based on the context provided by the relationship type.

The RelAtt Framework

The core contribution of this work is the RelAtt (Relation-aware Attention) layer. The framework operates in three distinct stages:

  1. Initial Embedding: For entities with textual attributes, the model utilizes a pre-trained BERT model to generate initial 768-dimensional feature vectors. In the absence of such attributes, random initialization is employed.
  2. Relation-Aware Attention Calculation: Unlike traditional attention mechanisms that rely solely on node features, RelAtt computes an attention coefficient $\alpha$ for a triplet $(h, r, t)$ by concatenating the features of the head entity, the relation, and the tail entity. A trainable weight vector then determines the relative importance of that specific neighbor.
  3. Weighted Feature Propagation: The model uses these calculated attention weights to scale the messages passed from neighbors during graph convolution. This builds upon the Relational GCN (RGCN) architecture but replaces uniform aggregation with a sophisticated, weighted attention-based sum.
The RelAtt mechanism computes attention weights for each neighbor based on the specific relation type.

The model is trained using a DistMult scoring function and cross-entropy loss, utilizing negative sampling to distinguish between valid and invalid triplets.

Experimental Evaluation

The authors evaluated RelAtt on three diverse datasets: WN18 (lexical relations), FB15k-237 (general world facts), and Comp (a proprietary IBM enterprise dataset). The Comp dataset is particularly noteworthy as it includes high-dimensional textual features for its 11,585 entities.

RelAtt consistently outperformed both translational models (DistMult) and standard graph neural networks (RGCN) across all benchmarks. On average, the model achieved a 2.8% improvement in Mean Reciprocal Rank (MRR). The results indicate that incorporating relation semantics into the attention mechanism allows the model to better navigate the complexities of both general-purpose and enterprise-specific graphs.

Link prediction performance comparison across three datasets, showing RelAtt's superiority in MRR and Hits@k metrics.

Entity Matching and Robustness

On the proprietary Comp dataset, RelAtt demonstrated significant gains in entity matching. At a low neighborhood threshold ($th=0.2$), RelAtt achieved a Hits@1 of 0.7380, substantially higher than RGCN's 0.6248. Furthermore, the model proved more stable than its counterparts when contextual information was sparse, suggesting that "smart" attention is vital when dealing with incomplete or sparse data.

Conclusion and Future Work

RelAtt represents a significant step forward in making graph neural networks context-aware. By integrating relation types into the attention mechanism, the model achieves superior performance in link prediction and entity matching, particularly in enterprise settings. Future research directions include addressing entity heterogeneity—treating different types of nodes (e.g., 'Company' vs. 'City') with specialized logic—and improving the scalability of the architecture for massive, billion-node knowledge graphs.

Summary updated

Abstract

Knowledge graph embedding methods learn embeddings of entities and relations in a low dimensional space which can be used for various downstream machine learning tasks such as link prediction and entity matching. Various graph convolutional network methods have been proposed which use different types of information to learn the features of entities and relations. However, these methods assign the same weight (importance) to the neighbors when aggregating the information, ignoring the role of different relations with the neighboring entities. To this end, we propose a relation-aware graph attention model that leverages relation information to compute different weights to the neighboring nodes for learning embeddings of entities and relations. We evaluate our proposed approach on link prediction and entity matching tasks. Our experimental results on link prediction on three datasets (one proprietary and two public) and results on unsupervised entity matching on one proprietary dataset demonstrate the effectiveness of the relation-aware attention.
BibTeX
@misc{sheikh2021knowledge,
  author = {Nasrullah Sheikh and Xiao Qin and Berthold Reinwald and Christoph Miksovic and Thomas Gschwind and Paolo Scotton},
  title = {{Knowledge Graph Embedding using Graph Convolutional Networks with Relation-Aware Attention}},
  year = {2021},
  eprint = {2102.07200},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2102.07200},
}
Figure