Knowledge Router: Learning Disentangled Representations for Knowledge Graphs

Shuai Zhang, Xi Rao, Yi Tay, Ce ZhangOpenReview 2021 · OpenReview

Abstract

The design of expressive representations of entities and relations in a knowledge graph is an important endeavor. While many of the existing approaches have primarily focused on learning from relational patterns and structural information, the intrinsic complexity of KG entities has been more or less overlooked. More concretely, we hypothesize KG entities may be more complex than we think, i.e., an entity may wear many hats and relational triplets may form due to more than a single reason. To this end, this paper proposes to learn disentangled representations of KG entities—a new method that disentangles the inner latent properties of KG entities. Our disentangled process operates at the graph level and a neighborhood mechanism is leveraged to disentangle the hidden properties of each entity. This disentangled representation learning approach is model agnostic and compatible with canonical KG embedding approaches. We conduct extensive experiments on several benchmark datasets, equipping a variety of models (DistMult, SimplE, and QuatE) with our proposed disentangling mechanism. Experimental results demonstrate that our proposed approach substantially improves performance on key metrics.

Introduction

In Knowledge Graphs (KGs), entities are often multifaceted. A single entity can play multiple roles depending on the context of its relations. For instance, a public figure might be represented as both an actor and a musician. Traditional KG embedding methods, such as TransE or DistMult, typically represent each entity as a single, holistic vector. This approach leads to entanglement, where distinct semantic roles are compressed into a single representation. Because these models often focus on individual triplets in isolation, they fail to capture the broader structural context necessary to distinguish these different facets. This lack of disentanglement can limit model robustness, interpretability, and predictive accuracy.

The Knowledge Router Framework

To address this, we introduce the Knowledge Router (KR), a model-agnostic framework designed to learn disentangled entity representations by leveraging graph-level structural information.

The Knowledge Router architecture: entity embeddings are split into K components, routed through a neighborhood mechanism for T iterations, and then concatenated.

The KR mechanism operates in three primary stages:

  1. Component Splitting: The initial entity embedding is partitioned into $K$ independent components, each intended to capture a specific latent "facet" or role of the entity.
  2. Neighborhood Routing: Inspired by Capsule Networks, KR uses an iterative routing mechanism to refine these components based on an entity's neighborhood. For each component, the model calculates similarity scores with the corresponding components of neighboring entities using scaled dot-product attention.
  3. Iterative Aggregation: Over $T$ iterations, the model aggregates information from neighbors that align with specific facets. This "routes" relevant structural information to the appropriate entity component, effectively clustering neighbors by the roles they interact with.

Crucially, KR acts as a "wrapper" that can be integrated with existing scoring functions like DistMult, SimplE, or QuatE without adding additional free parameters to the base models.

Experimental Evaluation

We evaluated the Knowledge Router on several benchmark datasets, including FB15k-237, WikiData, and the temporal datasets ICEWS14 and ICEWS05-15. Notably, WordNet was excluded from testing as its entities (synsets) are already semantically disentangled by design, making it an unsuitable test case for our hypothesis.

Quantitative Performance

KR consistently improved performance across all base models and metrics (MRR, Hit@1, Hit@3, Hit@10). For example, on the FB15k-237 dataset, KR-SimplE achieved a 19.2% improvement in MRR over the standard SimplE model. KR-QuatE also outperformed more complex architectures like RotatE and R-GCN on multiple benchmarks.

Qualitative Analysis: The "Many Hats" Hypothesis

A case study on the footballer Michael Rensing illustrates the model's ability to disentangle roles. As shown in the figure below, the routing mechanism successfully separated his "Club Player" facet (linking him to FC Bayern Munich) from his "National Team" facet (linking him to Germany's youth teams).

Case study of Michael Rensing: The heatmap shows how different neighbors are routed to distinct latent components, representing his roles in club vs. national football.

Discussion and Future Work

While KR provides significant gains, it introduces sensitivity to hyperparameters $K$ (number of components) and $T$ (routing iterations). Empirical results suggest that $K$ should generally not exceed the average degree of the graph to avoid representation degradation. Additionally, while the routing process is linear with respect to the number of neighbors, the $T$ iterations increase computational overhead compared to simple triplet-based methods. Future research will explore integrating temporal information into the disentanglement process for dynamic KGs.

Summary updated

Abstract

The design of expressive representations of entities and relations in a knowledge graph is an important endeavor. While many of the existing approaches have primarily focused on learning from relational patterns and structural information, the intrinsic complexity of KG entities has been more or less overlooked. More concretely, we hypothesize KG entities may be more complex than we think, ie, an entity may wear many hats and relational triplets may form due to more than a single reason. To this end, this paper proposes to learn disentangled representations of KG entities-a new method that disentangles the inner latent properties of KG entities. Our disentangled process operates at the graph level and a neighborhood mechanism is leveraged to disentangle the hidden properties of each entity. This disentangled representation learning approach is model agnostic and compatible with canonical KG embedding approaches. We conduct extensive experiments on several benchmark datasets, equipping a variety of models (DistMult, SimplE, and QuatE) with our proposed disentangling mechanism. Experimental results demonstrate that our proposed approach substantially improves performance on key metrics.
BibTeX
@misc{zhang2021knowledge,
  author = {Shuai Zhang and Xi Rao and Yi Tay and Ce Zhang},
  title = {{Knowledge Router: Learning Disentangled Representations for Knowledge Graphs}},
  year = {2021},
  url = {https://openreview.net/forum?id=rZoFyLD0Rd2},
  howpublished = {OpenReview},
}
Figure