Path-enhanced Pre-trained Language Model for Knowledge Graph Completion

Hao Wang, Dandan Song, Zhijing Wu, Yuhang Tian, Pan YangEMNLP 2025

TL;DR

This paper introduces PEKGC, a novel approach to knowledge graph completion (KGC) that leverages both direct triple information and longer, multi-hop paths to improve accuracy and interpretability. By combining triple-level and path-level predictions with a unique fusion strategy, PEKGC significantly boosts performance on benchmark KGs, offering a more robust and understandable way to fill in missing facts.

What Problem Does This Paper Address?

Knowledge graphs (KGs) are powerful tools for organizing information, but they are often incomplete. Knowledge Graph Completion (KGC) aims to predict these missing links (facts). Existing methods, especially those using pre-trained language models (PLMs), often rely on "flat" representations of triples (e.g., "entity-relation-entity") or text descriptions. While effective to a degree, this approach has limitations:

  • Lack of Long-Term Relational Information: It struggles to capture complex, multi-hop relationships that span across several entities and relations.
  • Poor Interpretability: The reasoning process behind predictions is often opaque, making it hard to trust or understand why a particular completion was suggested.
  • Limited Expressiveness: Focusing only on immediate neighbors can miss crucial structural information within the KG.

This paper tackles these issues by proposing a method that explicitly incorporates both direct triple information and longer reasoning paths, aiming for higher accuracy and more interpretable results.

Methods

The core of PEKGC is a multi-view generation paradigm built upon a sequence-to-sequence (Seq2Seq) architecture, similar to models like T5. It tackles KGC from two primary perspectives:

  1. Triple-Level Generation: This standard approach encodes a query (e.g., "Tu Youyou, Nationality, ?") and predicts the missing entity. To enrich context, it incorporates entity descriptions.
  2. Path-Level Generation: This novel aspect explicitly models multi-hop reasoning paths. For a query, it generates not only the target entity but also the sequence of relations and intermediate entities that form a reasoning chain. This provides a more interpretable explanation for the prediction.

Key innovations include:

  • Multi-View Generation: Simultaneously generating predictions at both triple and path levels allows for cross-validation and leveraging different types of KG information.
  • Neighbor Selector Module: A pre-trained module that filters relevant neighbor triples around the query entity. This helps incorporate local structural information without overwhelming the model.
  • KG Soft Prompts & Position Soft Prompts: These are introduced to help the model distinguish between KG knowledge and textual knowledge, and to mark the positional relationships within a triple.
  • Fact-Level Re-evaluation: A mechanism to re-evaluate the confidence scores of candidate entities generated by both triple and path levels, particularly for "hard" predictions where the two views disagree significantly.
  • Heuristic Fusion Ranking Strategy: A sophisticated strategy to combine the results from triple-level and path-level predictions. It categorizes predictions into "consistent" and "hard" cases to decide whether to rely on path-level ranks, re-evaluate, or use a combination, aiming to balance performance and efficiency.

Datasets

The paper evaluates PEKGC on two widely used benchmark datasets for knowledge graph completion:

  • WN18RR: Derived from WordNet, focusing on relational links.
  • FB15k-237: A subset of Freebase, known for its challenging link prediction task.

No new datasets were introduced in this paper.

Key Insights and Ideas

  • Paths as Explanations: The path-level generation is crucial not just for accuracy but also for interpretability. Seeing the reasoning chain (e.g., "Tu Youyou - Born In - Zhejiang - Located In - China") makes the prediction much more understandable than just outputting "China."
  • Complementary Views: Combining direct triple information with longer reasoning paths is more powerful than relying on either alone. The path-level view helps overcome the limitations of local information and provides context for ambiguous cases.
  • Handling Disagreement: The "hard prediction" scenario, where triple-level and path-level predictions diverge, is a critical area. The fact-level re-evaluation and fusion ranking strategy are designed to intelligently handle these disagreements, leading to better overall performance.
  • Neighbor Selection Matters: Incorporating relevant neighboring triples, even if indirectly, helps enrich the model's understanding of local KG structure.

Key Results

PEKGC demonstrates strong performance across both WN18RR and FB15k-237 datasets.

  • Improved Accuracy: PEKGC achieves state-of-the-art results among PLM-based KGC models, notably improving the Hits@1 metric by 6.4% on WN18RR and 6.7% on FB15k-237 compared to previous best models.
  • Efficiency Trade-off: While PEKGC's training time is slightly higher than simpler models like KG-S2S (due to path and neighbor processing), the performance gains (e.g., average Hits@1 improvement of 22.2% and 30.7%) justify the increased runtime.
  • Module Effectiveness: Ablation studies confirm that each proposed module (path-level generation, neighbor selector, soft prompts, fusion ranking) contributes significantly to the overall performance. Removing any of them leads to a noticeable drop in accuracy.
  • LLM Comparison: PEKGC outperforms purely LLM-based methods on WN18RR, highlighting the continued value of specialized pre-training and architectural design for KGC tasks, even as LLMs advance.

Limitations and Future Work

The authors acknowledge a few limitations:

  • Diversity of Generated Results: The model can sometimes produce a high number of identical or homogeneous outputs, which might affect its robustness and generalization, especially for 1-N or N-N triple prediction scenarios.
  • Decoder Bias: Decoders may tend to favor generating seen entities, potentially limiting flexibility and generalization.

For future work, the authors aim to address these challenges, focusing on improving the diversity and generalization capabilities of PLM-based KGC models.

Summary updated

BibTeX
@inproceedings{wang2025path,
  author = {Hao Wang and Dandan Song and Zhijing Wu and Yuhang Tian and Pan Yang},
  title = {{Path-enhanced Pre-trained Language Model for Knowledge Graph Completion}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure