COSIGN: Contextual Facts Guided Generation for Knowledge Graph Completion

Jinpeng Li, Hang Yu, Xiangfeng Luo, Qian Liu

Abstract

Knowledge Graph Completion (KGC) is essential for addressing the inherent incompleteness of large-scale knowledge bases. While generative models (GMs) have emerged as a scalable alternative to traditional embedding-based methods, they frequently suffer from context sensitivity, leading to hallucinations when processing unorganized graph data. We introduce COSIGN (COntextual FactS GuIded GeneratioN), a novel framework that improves KGC by systematically collecting and organizing contextual facts into coherent narratives. By distilling the organizational logic of large language models (LLMs) into smaller, efficient models, COSIGN achieves state-of-the-art performance across static, temporal, and few-shot graph benchmarks.

Introduction

Knowledge Graphs (KGs) represent facts as networks of entities and relations, such as (Barack Obama, visit, France). Despite their utility, KGs are often incomplete. Predicting missing links, known as Knowledge Graph Completion (KGC), is a fundamental task in AI. Recent shifts toward Generative Models (GMs) like T5 or GPT treat KGC as a text-to-text problem, leveraging their language understanding capabilities.

However, GMs are highly sensitive to the quality of input context. Providing a "messy" collection of scattered facts often results in incorrect predictions or hallucinations. As illustrated in Figure 1, the difference between organized and scattered facts is the difference between a correct result and a partially rational or even ridiculous one. The core challenge lies in providing the model with relevant context organized to facilitate reasoning.

The impact of contextual organization on model reasoning.

Methodology: The COSIGN Framework

COSIGN addresses the context sensitivity problem through a three-stage pipeline:

1. Contextual Facts Collector (CFC)

The CFC module emulates human-like retrieval by pruning the local neighborhood of a query and using a generative filter to select the most relevant facts. It employs a "positive-set confidence advantage" loss function to distinguish between semantically similar but distinct facts. Finally, it identifies the shortest paths between these facts to establish a logical chain of information.

2. Contextual Facts Organizer (CFO)

Raw graph triples are often difficult for generative models to parse effectively. The CFO module uses knowledge distillation to bridge this gap. A powerful LLM (e.g., GPT-3.5) is prompted to transform scattered facts into a coherent, summarized narrative. A smaller model (SLM) is then trained to mimic this organizational ability, providing high-quality context at a fraction of the computational cost.

3. Inference Generator

The final stage combines the query with the organized context to predict the missing entity. To eliminate hallucinations, the authors implement prefix constraints, ensuring the decoder only generates valid entity names present in the KG.

The COSIGN architecture, featuring the Collector, Organizer, and Inference Generator.

Experimental Evaluation

COSIGN was evaluated on five benchmark datasets across three KGC paradigms:

  • Static KGC: WN18RR, FB15K-237, and FB15K-237N.
  • Temporal KGC: ICEWS14, requiring reasoning over time-stamped facts.
  • Few-Shot KGC: NELL-One, testing the ability to predict relations with minimal training data.

Key Results and Insights

The experimental results demonstrate that COSIGN consistently outperforms both traditional embedding-based models and previous state-of-the-art generative models.

  • Accuracy Boost: On WN18RR, COSIGN improved Hit@1 by 7.9% over the previous best generative model, KG-S2S.
  • Temporal Reasoning: The framework achieved a 5.7% relative improvement in Hit@1 on the ICEWS14 dataset, showcasing its superior handling of time-sensitive data.
  • Efficiency: COSIGN maintains low latency, adding only approximately 149ms compared to simpler baselines. Furthermore, it reaches near-optimal performance using only 40% of the training data, indicating the effectiveness of the distilled organizational logic.
  • Coherence is King: The primary insight is that the presentation of data is as critical as the data itself. Converting triples into concise summaries significantly reduces model confusion.

Conclusion and Future Work

COSIGN demonstrates that intelligent context organization is a powerful tool for enhancing generative KGC. While highly effective, the model's performance is tied to the richness of the available context. In sparse graphs, such as those found in few-shot scenarios like NELL-One, the collector may struggle to find sufficient facts. Future research will focus on mitigating "semantic sparsity" by integrating external knowledge sources and improving reasoning strategies for data-limited environments.

Summary updated

Abstract

Knowledge graph completion (KGC) aims to infer missing facts based on existing facts within a KG. Recently, research on generative models (GMs) has addressed the limitations of embedding methods in terms of generality and scalability. However, GM-based methods are sensitive to contextual facts on KG, so the contextual facts of poor quality can cause GMs to generate erroneous results. To improve the performance of GM-based methods for various KGC tasks, we propose a COntextual FactS GuIded GeneratioN (COSIGN) model. First, to enhance the inference ability of the generative model, we designed a contextual facts collector to achieve human-like retrieval behavior. Second, a contextual facts organizer is proposed to learn the organized capabilities of LLMs through knowledge distillation. Finally, the organized contextual facts as the input of the inference generator to generate missing facts. Experimental results demonstrate that COSIGN outperforms state-of-the-art baseline techniques in terms of performance.
BibTeX
@inproceedings{li2024cosign,
  author = {Jinpeng Li and Hang Yu and Xiangfeng Luo and Qian Liu},
  title = {{COSIGN: Contextual Facts Guided Generation for Knowledge Graph Completion}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.93},
}
Figure