Text-Enhanced Representation Learning for Knowledge Graph

Zhigang Wang, Juan-Zi LiIJCAI 2016

Abstract

The Text-Enhanced Knowledge Embedding (TEKE) framework significantly improves Knowledge Graph (KG) representation by incorporating rich contextual information from external text corpora. By utilizing entity co-occurrence networks to dynamically adjust embeddings, TEKE effectively addresses the limitations of translation-based models in handling complex many-to-many relations and data sparsity.

Introduction

Knowledge Graphs represent facts as triples: (head entity, relation, tail entity), such as (Avatar, directed_by, James Cameron). Standard "translation-based" models like TransE attempt to learn vectors where $\mathbf{h} + \mathbf{r} \approx \mathbf{t}$. While elegant, these models face two major hurdles:

  1. Complex Relations: In 1-to-N or N-to-N relations (e.g., one director making many movies), models like TransE struggle because they attempt to map the same head and relation to multiple distinct tail entities, leading to "blurred" embeddings.
  2. Data Sparseness: Many KGs are sparse, meaning there are insufficient structural links to learn high-quality representations for every entity.

TEKE addresses these issues by using unstructured text as the "semantic glue" to enhance structural embeddings.

Methodology

The TEKE framework is a flexible architecture that can be integrated into existing models like TransE, TransH, or TransR. The process follows four main steps:

The TEKE framework pipeline, illustrating the flow from raw text and KG to final embeddings.
  1. Entity Annotation: Entities in the KG are linked to a large text corpus (Wikipedia) using entity linking tools.
  2. Textual Context Embedding: The system builds a co-occurrence network of entities and words. Using word2vec, it calculates Pointwise Context (the average representation of surrounding terms) and Pairwise Context (the shared context between a head and a tail entity).
  3. Representation Modeling: Instead of static vectors, TEKE defines "enhanced" representations. The head entity $\hat{\mathbf{h}}$ is a linear transformation of its textual context plus its original embedding, while the relation $\hat{\mathbf{r}}$ is adjusted based on the pairwise context of the entities involved.
  4. Training: The model is optimized using a margin-based ranking loss, encouraging correct triples to have higher scores than corrupted ones.

Experimental Evaluation

TEKE was evaluated on standard benchmarks including WN18, WN11, FB15K, and FB13, with Wikipedia as the external corpus. To test robustness against sparsity, the authors also created smaller subsets (FB3K, FB6K, FB9K).

Performance comparison of TEKE-enhanced models against baseline translation models on WN18 and FB15K datasets.

Key Results

  • Link Prediction: TEKE consistently outperformed state-of-the-art baselines. On the FB15K dataset, TEKE improved the Hits@10 (Filter) from 47.1% to 67.6% for the base TransE model.
  • Complex Relations: The most dramatic gains occurred in N-to-1 and 1-to-N relations. For instance, in predicting head entities for N-to-1 relations, TEKE improved performance by nearly 20% over standard models.
  • Handling Sparseness: In sparse graph tests (FB3K), TEKE achieved a significantly better Mean Rank (94.9) compared to TransE (102.7), proving that textual context compensates for missing structural links.

Discussion and Conclusion

The core innovation of TEKE is the use of Dynamic Embeddings. By allowing an entity's representation to "shift" based on its textual context, the model gains the flexibility needed to handle complex relations. While TEKE_R (the TransR-based version) involves higher computational complexity, the framework successfully bridges the gap between structured knowledge and unstructured text. Future work will focus on incorporating multi-step knowledge reasoning and refining accuracy for 1-to-1 relations.

Summary updated

BibTeX
@inproceedings{wang2016text,
  author = {Zhigang Wang and Juan-Zi Li},
  title = {{Text-Enhanced Representation Learning for Knowledge Graph}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
}
Figure