Joint Pre-Encoding Representation and Structure Embedding for Efficient and Low-Resource Knowledge Graph Completion

Chenyu Qiu, Pengjiang Qian, Chuang Wang, Jian Yao, Li Liu, Wei Fang, Eddie EddieEMNLP 2024 · DOI

Introduction

Knowledge Graphs (KGs) such as Wikidata and WordNet serve as critical backbones for various AI applications, yet they are often incomplete. Knowledge Graph Completion (KGC) aims to predict missing links within these networks. Traditionally, researchers have faced a trade-off: fast but semantically shallow embedding-based models versus accurate but computationally expensive description-based models.

Researchers have recently introduced PEMLM (Pre-Encoded Masked Language Model), a framework designed to bridge this gap. By pre-encoding textual descriptions and employing a classification-based ranking strategy, PEMLM achieves state-of-the-art results while increasing inference speed by 30x and reducing training memory usage by 60%.

The Efficiency Gap in KGC

Existing KGC methods generally fall into two categories:

  1. Embedding-based models: These map entities into vector spaces based on graph structure. They are efficient but ignore the rich semantic context found in entity descriptions.
  2. Description-based models: These utilize Pre-trained Language Models (PLMs) like BERT to process text. While highly accurate, they suffer from a "resource wall." Models like KG-BERT require significant inference time because they must process long text sequences for every possible candidate entity—an $O(N)$ operation that becomes a bottleneck as the number of entities grows.

PEMLM addresses this by making description-based KGC efficient enough for low-resource environments without sacrificing the semantic depth provided by PLMs.

The PEMLM Framework

The authors propose two variants: PEMLM and PEMLM-F (Fusion).

Pre-Encoding and Triplet Encoding

Instead of processing raw text during every training step, PEMLM uses a frozen BERT model to encode entity and relation descriptions into fixed semantic vectors before training begins. This shifts the computational burden from real-time text processing to lighter vector processing.

The PEMLM architecture, highlighting the pre-encoding of descriptions followed by a classification-based training phase.

The model treats KGC as a Masked Language Modeling (MLM) task. For a triple $(h, r, ?)$, the pre-encoded vectors for the head ($h$) and relation ($r$) are fed into a "Triplet Encoder" alongside a [MASK] token. Unlike traditional models that iterate through every candidate, PEMLM uses a single classification layer over the [MASK] token's output to predict the tail entity in $O(1)$ time.

Structural Fusion (PEMLM-F)

To capture the geometric "shape" of the graph, the PEMLM-F variant includes a learnable fusion module. This module integrates semantic vectors with structural embeddings (inspired by TransE), specifically improving the model's ability to handle complex "1-N" relations where one head and relation can point to multiple valid tails.

Experimental Results

The model was evaluated on three standard benchmarks: WN18RR (lexical), FB15k-237 (general knowledge), and UMLS (medical).

Performance comparison of PEMLM and PEMLM-F against embedding-based, description-based, and joint-based models across three benchmarks.

Key Findings:

  • State-of-the-Art Performance: On WN18RR, PEMLM-F achieved a +5.4% improvement in Mean Reciprocal Rank (MRR) and a +6.4% increase in Hits@1 compared to previous leading models.
  • Efficiency Gains: The model is 30 times faster during inference than KG-BERT and reduces training memory requirements by approximately 60%, enabling training on consumer-grade hardware.
  • Semantic vs. Structure: Results indicate that while text is excellent for defining an entity, structural data is superior for understanding relational patterns. The fusion of these signals is particularly effective for complex relations.

Discussion and Future Directions

While PEMLM offers significant efficiency gains, its performance is inherently tied to the quality of available text descriptions. In datasets where descriptions are sparse or absent, the model's semantic advantage is limited. Furthermore, aligning high-dimensional graph structures with pre-encoded text representations remains a complex challenge.

Future work will focus on improving the alignment of these disparate embedding spaces and exploring the integration of even larger language models (LLMs) while maintaining the efficiency of the pre-encoding strategy.

Summary updated

BibTeX
@inproceedings{qiu2024joint,
  author = {Chenyu Qiu and Pengjiang Qian and Chuang Wang and Jian Yao and Li Liu and Wei Fang and Eddie Eddie},
  title = {{Joint Pre-Encoding Representation and Structure Embedding for Efficient and Low-Resource Knowledge Graph Completion}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.851},
}
Figure