Decentralized Knowledge Graph Representation Learning

Lingbing Guo, Weiqing Wang, Zequn Sun, Chenghao Liu, Wei Hu

Introduction

Knowledge Graphs (KGs) serve as the backbone for numerous AI applications, from search engines to recommendation systems, by storing facts as interconnected triplets (e.g., Tim Berners-Lee -> founded -> W3C). While Graph Neural Networks (GNNs) have become the standard for learning entity embeddings, they face a significant hurdle: the "open-world" problem. In dynamic real-world environments, new entities are constantly added, yet standard GNNs rely on pre-learned, transductive embeddings that fail when encountering unseen nodes.

This paper introduces decentRL, a framework featuring the Decentralized Attention Network (DAN). By redefining how entities are represented—moving from self-centered embeddings to neighborhood-derived contexts—decentRL enables robust performance in both conventional and open-world KG scenarios.

What Problem Does This Paper Address?

Traditional KG embedding methods are primarily transductive. They learn a specific vector for every entity seen during training. When a new entity appears in an open-world setting, its self-embedding is typically initialized randomly. This random initialization acts as noise, disrupting the GNN's aggregation process and leading to poor performance on inductive tasks like link prediction for new nodes.

Comparison of Knowledge Graph and Word Embedding paradigms, highlighting the shift from 1-to-1 to N-to-1 models.

The authors challenge the necessity of an entity's self-embedding for encoding its relational identity. They draw a parallel to the Continuous Bag-of-Words (CBOW) model in Natural Language Processing, where a central word is predicted entirely from its surrounding context. If a word can be defined by the company it keeps, can an entity be defined solely by its neighbors?

Methods: The decentRL Framework

To address the limitations of transductive models, the authors propose two core innovations within the decentRL framework:

1. Decentralized Attention Network (DAN)

Unlike standard Graph Attention Networks (GATs) that use a central entity's embedding to compute attention over its neighbors, DAN completely removes the central entity from the equation. It aggregates neighbor embeddings to create a "neighbor context" vector, which then serves as the query to score and aggregate those same neighbors. This "collective voting" mechanism ensures that the resulting embedding is derived exclusively from the graph topology, making it naturally resilient to the noise of unseen entities.

The CBOW model analogy: predicting a central entity (W3C) using only its surrounding context (neighbors).

2. Self-Distillation Training

If the central entity's input is ignored during the forward pass, the model needs a way to learn its identity. The authors employ a teacher-student self-distillation process. The ignored input embedding acts as a "teacher," providing a target for the neighbor-generated "student" embedding. By maximizing the mutual information between the two using an InfoNCE contrastive loss, the network learns to reconstruct an entity's core identity purely from its neighborhood context.

Experimental Evaluation

The authors evaluated decentRL across two primary tasks: Entity Alignment and Entity Prediction.

  • Conventional Benchmarks: On standard datasets like DBP15K, decentRL achieved state-of-the-art results, reaching a Hits@1 of 0.589 in entity alignment, significantly outperforming the AliNet baseline (0.539).
  • Open-World Scenarios: To test inductive capabilities, the authors created new benchmarks where 20% of entities were hidden during training. In these challenging settings, decentRL's advantage grew. For example, on the open-world FB15K-237 dataset, it achieved an MRR of 0.255, substantially higher than existing open-world methods like LAN (0.164).

Key Insights and Efficiency

A critical finding of the study is that the benefits of decentralized attention scale with an entity's degree. While isolated entities with few neighbors gain little, the performance of DAN improves rapidly as the neighborhood context grows richer.

Furthermore, decentRL proves to be highly efficient. The self-distillation mechanism provides a powerful training signal that leads to faster convergence. On average, decentRL required only 55 epochs to converge, compared to 280 epochs for AliNet, resulting in a significant reduction in total training time.

Conclusion and Future Work

By shifting the focus from individual entity embeddings to decentralized neighborhood contexts, decentRL provides a robust solution for the open-world Knowledge Graph challenge. While the model currently relies heavily on graph topology, future work may involve integrating multi-modal features like text and images. The success of decentralized attention and self-distillation in KGs suggests potential applications in other domains, such as social network analysis and molecular property prediction.

Summary updated

Abstract

Knowledge graph (KG) representation learning methods have achieved competitive performance in many KG-oriented tasks, among which the best ones are usually based on graph neural networks (GNNs), a powerful family of networks that learns the representation of an entity by aggregating the features of its neighbors and itself. However, many KG representation learning scenarios only provide the structure information that describes the relationships among entities, causing that entities have no input features. In this case, existing aggregation mechanisms are incapable of inducing embeddings of unseen entities as these entities have no pre-defined features for aggregation. In this paper, we present a decentralized KG representation learning approach, decentRL, which encodes each entity from and only from the embeddings of its neighbors. For optimization, we design an algorithm to distill knowledge from the model itself such that the output embeddings can continuously gain knowledge from the corresponding original embeddings. Extensive experiments show that the proposed approach performed better than many cutting-edge models on the entity alignment task, and achieved competitive performance on the entity prediction task. Furthermore, under the inductive setting, it significantly outperformed all baselines on both tasks.
BibTeX
@article{guo2024decentralized,
  author = {Lingbing Guo and Weiqing Wang and Zequn Sun and Chenghao Liu and Wei Hu},
  title = {{Decentralized Knowledge Graph Representation Learning}},
  journal = {Knowledge-Based Systems},
  year = {2024},
  doi = {10.1016/j.knosys.2024.111582},
}
Figure