Dolores: Deep Contextualized Knowledge Graph Embeddings

Haoyu Wang, Vivek Kulkarni, William Yang WangAKBC 2018 · OpenReview

TL;DR

This paper introduces DOLORES, a novel method for learning "deep" and "contextualized" knowledge graph embeddings by treating knowledge graphs as sequences of entity-relation chains, inspired by language modeling. DOLORES significantly improves performance on various knowledge graph prediction tasks by providing richer, multi-layered representations of entities and relations.

What Problem Does This Paper Address?

Knowledge graphs (KGs) are powerful tools for organizing world knowledge, but representing them solely as discrete triples (entityA, relationX, entityB) limits our ability to infer missing relationships and understand nuanced entity/relation contexts. Traditional KG embedding methods often rely on shallow models and struggle to capture the rich contextual information embedded within the structure of the graph. This paper addresses the challenge of learning more effective KG embeddings that can capture both context-independent and context-dependent meanings of entities and relations, leading to better performance on downstream tasks.

Methods

DOLORES tackles this problem by drawing inspiration from recent advances in deep learning for natural language processing, specifically the use of Bi-Directional LSTMs for learning contextual word embeddings. The core idea is to represent a knowledge graph not as a collection of triples, but as a collection of "entity-relation chains" generated by random walks.

The DOLORES method has two main components:

  1. Path Generator: This component generates a large corpus of entity-relation chains by performing random walks on the knowledge graph. These walks are guided by parameters (similar to those in Node2Vec) that control the sampling strategy, effectively creating "sentences" from the graph structure.
  2. Learner: This component is a deep neural model, specifically a Bi-Directional LSTM, that processes these entity-relation chains. It learns embeddings for entities and relations by considering their context within these chains. The model learns multiple representations for each entity/relation at different layers of the LSTM, capturing varying levels of abstraction. These learned representations are a weighted linear combination of the internal states of the LSTM, allowing for both context-independent (from the initial embedding layer) and context-dependent (from LSTM layers) cues.

Datasets

The paper evaluates DOLORES on several standard knowledge graph datasets:

  • FB15K-237: Used for link prediction tasks.
  • WN11 and FB13: Used for triple classification tasks.
  • A subset of Freebase enriched with information from CLUEWEB: Used for missing relation type prediction.

The authors do not introduce new benchmarks but rather demonstrate the effectiveness of DOLORES by integrating it with existing state-of-the-art models on these established datasets.

Key Insights and Ideas

  • KG as Sequences: The fundamental shift in perspective is viewing knowledge graphs not as static sets of facts but as dynamic sequences of entity-relation chains, analogous to sentences in language. This allows for the application of powerful sequential modeling techniques.
  • Deep, Multi-Layered Embeddings: By using a deep Bi-Directional LSTM, DOLORES learns embeddings that are not just a single vector but a collection of representations from different layers. This allows for capturing different levels of abstraction, from general entity types to specific contextual meanings.
  • Contextualization is Key: The paper highlights how context-dependent embeddings can significantly improve performance. For example, embeddings can differentiate between entities based on their nationality or the places they lived, even if they share other attributes.
  • "Drop-in" Replacement: A crucial insight is that DOLORES embeddings can be easily incorporated into existing predictive models that use embedding layers, leading to significant performance gains without requiring a complete redesign of the downstream model.

Key Results

DOLORES embeddings consistently improve the performance of state-of-the-art models across various knowledge graph prediction tasks:

  • Link Prediction: Achieved improvements of up to 7.9% in tail entity prediction and 3.10% in head entity prediction on the FB15K-237 dataset, as measured by MRR.
  • Triple Classification: Showed a slight but notable improvement on the CONVKB model, increasing the average performance from 88.20 to 88.40.
  • Missing Relation Type Prediction: Demonstrated a significant improvement of 9.5% (from 71.74 to 74.42) on the mean average precision (MAP) when incorporated into the RNN-Path-entity model.

Qualitatively, the paper shows that DOLORES embeddings can better distinguish between specific and generic entities, leading to more accurate predictions, especially when the head entity is specific.

Limitations and Future Work

  • Limitations:
    • The paper notes that DOLORES performs best when head entities are specific and may struggle with very broad or generic head entities.
  • While improvements are significant, the absolute gains in triple classification are smaller because the baseline models are already very strong.
  • Future Work:
    • Investigating the utility of individual layers' embeddings in specific learning tasks, drawing parallels to transfer learning in computer vision.
  • Exploring the effectiveness of these learned embeddings in transfer-learning scenarios across different models and datasets.
  • Investigating the integration of DOLORES embeddings with natural language processing models for tasks like relation extraction and named entity recognition.
  • The paper proposes a new paradigm of viewing KGs as paths and learning multiple representations, opening avenues for further research into deep and contextualized KG modeling.

Summary updated

Abstract

We introduce a new method DOLORES for learning knowledge graph embeddings that effectively captures contextual cues and dependencies among entities and relations. First, we note that short paths on knowledge graphs comprising of chains of entities and relations can encode valuable information regarding their contextual usage. We operationalize this notion by representing knowledge graphs not as a collection of triples but as a collection of entity-relation chains, and learn embeddings for entities and relations using deep neural models that capture such contextual usage. In particular, our model is based on Bi-Directional LSTMs and learn deep representations of entities and relations from constructed entity-relation chains. We show that these representations can very easily be incorporated into existing models to significantly advance the state of the art on several knowledge graph prediction tasks like link prediction, triple classification, and missing relation type prediction (in some cases by at least 9.5%).
BibTeX
@inproceedings{wang2018dolores,
  author = {Haoyu Wang and Vivek Kulkarni and William Yang Wang},
  title = {{Dolores: Deep Contextualized Knowledge Graph Embeddings}},
  booktitle = {AKBC},
  year = {2018},
  url = {https://openreview.net/forum?id=RUFl9sDrw-z},
}
Figure