Abstract
Large Language Models (LLMs) have recently emerged as a powerful paradigm for Knowledge Graph Completion (KGC), offering strong reasoning and generalization capabilities beyond traditional embedding-based approaches. However, existing LLM-based methods often struggle to fully exploit structured semantic representations, as the continuous embedding space of pretrained KG models is fundamentally misaligned with the discrete token space of LLMs. This discrepancy hinders effective semantic transfer and limits their performance. To address this challenge, we propose ReaLM, a novel and effective framework that bridges the gap between KG embeddings and LLM tokenization through the mechanism of residual vector quantization. ReaLM discretizes pretrained KG embeddings into compact code sequences and integrates them as learnable tokens within the LLM vocabulary, enabling seamless fusion of symbolic and contextual knowledge. Furthermore, we incorporate ontology-guided class constraints to enforce semantic consistency, refining entity predictions based on class-level compatibility. Extensive experiments on two widely used benchmark datasets demonstrate that ReaLM achieves state-of-the-art performance, confirming its effectiveness in aligning structured knowledge with large-scale language models.
Introduction
Knowledge Graph Completion (KGC) is the task of predicting missing links in a structured knowledge base—for instance, identifying the spouse of a specific entity. While traditional methods utilize geometric embeddings (e.g., RotatE) to represent entities in a continuous vector space, they often lack the deep reasoning and linguistic context inherent in Large Language Models (LLMs). Conversely, LLMs operate in a discrete token space, creating a significant "impedance mismatch" when integrating structured graph data.
Existing attempts to bridge this gap face three primary hurdles: the fundamental mismatch between continuous vectors and discrete tokens, the "vocabulary explosion" caused by millions of unique entities, and the risk of hallucination when generating entity names as plain text.
Methodology: The ReaLM Framework
ReaLM (Residual Quantization Bridging Knowledge Graph Embeddings and Large Language Models) introduces a multi-stage pipeline to align these disparate spaces:
- Semantic Extraction: High-quality continuous embeddings are first generated for all entities using a traditional model like RotatE.
- Residual Vector Quantization (RVQ): This core component represents each entity as a sequence of 32 "codewords." By using a multi-stage process where each stage captures the residual error of the previous one, a small codebook (e.g., 1,000 tokens) can represent a vast number of unique entities without vocabulary explosion.
- Token Integration: These codewords are added to the LLM's vocabulary as learnable tokens. The model is then fine-tuned using Low-Rank Adaptation (LoRA) to process these sequences.
- Ontology-Guided Refinement: To ensure semantic consistency, ReaLM predicts the entity's class (e.g., "Person") alongside its codewords, using this as a filter to prevent type errors.
Experimental Evaluation
The framework was evaluated on standard benchmarks WN18RR (lexical relations) and FB15k-237 (real-world facts), including ontology-augmented versions (WN18RRO and FB15k-237O).
Key Results
- Performance: ReaLM achieved state-of-the-art results, with an MRR of 0.608 on WN18RR, representing a 1.6%–3.5% improvement over previous LLM-based baselines.
- Precision: The model demonstrated high accuracy in exact entity prediction (Hits@1), reaching 0.575 on WN18RR.
- Efficiency: By utilizing RVQ, ReaLM successfully represented over 40,000 entities using only 1,000 additional tokens, maintaining computational efficiency.
Conclusion and Future Work
ReaLM effectively bridges the gap between continuous KG embeddings and discrete LLM tokens. By compressing semantic meaning into hierarchical codewords, it enables LLMs to reason over structured knowledge with high precision. Future work will focus on reducing the computational cost of training—currently requiring significant GPU hours—and extending the framework to dynamic, industrial-scale knowledge graphs.
Summary updated