SCE: Semantic Consistency Enhanced Reinforcement Learning for Multi-Hop Knowledge Graph Reasoning

Yanwen Huang, Yao Liu, Qiao Liu, Rui Hou, Tingting DaiEMNLP 2025

TL;DR

This paper introduces SCE, a novel reinforcement learning framework for multi-hop knowledge graph reasoning that tackles the "spurious path" problem. By incorporating semantic consistency between intermediate and target triples into the reward function, SCE guides agents to discover more interpretable and accurate reasoning paths, outperforming existing methods.

What Problem Does This Paper Address?

Knowledge Graphs (KGs) are powerful for tasks like question answering, but they are often incomplete. Multi-hop reasoning aims to infer missing facts by chaining together multiple relationships. Reinforcement learning (RL) has been effective for this, but a major challenge is "spurious paths": incorrect reasoning chains that accidentally lead to the correct answer. This happens because traditional RL rewards often focus solely on the final answer's accuracy, ignoring the quality of the reasoning steps. Existing methods try to fix this with external rules, but they often overlook the internal semantic coherence of the reasoning path itself. This paper addresses the need for a method that explicitly considers the semantic consistency of the entire reasoning path, not just the final outcome.

Methods

The core of this paper is the Semantic Consistency Enhanced Reinforcement Learning (SCE) framework. It builds upon a standard RL formulation for multi-hop reasoning, where an agent navigates a KG to find a target entity for a given query.

The novelty lies in how SCE enhances the reward function:

  • Semantic Consistency Score (SC): This score quantifies how semantically coherent a reasoning path is. It's a combination of two main components:
  • Relation Consistency (RC): Measures how well the relations in the reasoning path align with the query relation. This includes:
  • Local Relation Similarity (LRS): Cosine similarity between individual path relations and the query relation.
  • Global Relation Closeness (GRC): Ensures that each step in the path progressively reduces the distance to the target entity.
  • Entity Consistency (EC): Measures the semantic coherence between intermediate entities and the target entity. This includes:
  • Local Entity Similarity (LES): Cosine similarity between intermediate entities and the target entity.
  • Global Entity Closeness (GEC): Ensures that entities in the path progressively move closer to the target entity.
  • Reasonable Reward (Rr): The SC score is integrated into the reward function to encourage the agent to favor semantically consistent paths, especially when the answer is correct.
  • Final Reward (R): This combines the accuracy reward (whether the final answer is correct) with the reasonable reward, ensuring both accuracy and interpretability are prioritized.

The paper uses the REINFORCE algorithm for optimization and pre-trained embeddings (like ConvE) to provide soft rewards, mitigating sparse feedback.

Datasets

The authors evaluated their method on five widely used knowledge graph datasets:

  • UMLS
  • Kinship
  • FB15k-237
  • WN18RR
  • WD15K

No new benchmarks were introduced; these are standard datasets for KG reasoning tasks.

Key Insights and Ideas

  • Spurious paths are a real problem: The paper clearly illustrates how RL agents can find correct answers through nonsensical reasoning chains, highlighting the need for more than just accuracy-based rewards.
  • Internal consistency matters: The core idea is that a good reasoning path isn't just about the final answer but also about the semantic coherence of each step. SCE formalizes this by measuring consistency between intermediate and target triples.
  • Relation and Entity consistency are complementary: The paper shows that both relation-level and entity-level consistency are crucial. Neither alone is sufficient, but together they provide a robust measure of path quality.
  • Semantic consistency correlates with human interpretability: The quantitative analysis in Appendix A strongly suggests that higher semantic consistency scores align with human judgments of path interpretability.
  • Efficiency advantage: SCE achieves better interpretability and performance with lower computational overhead compared to methods that rely on external rules.

Key Results

  • Performance: SCE consistently achieved the best performance across most datasets, showing improvements in Hit@1 and MRR over strong baselines like PS-Agent. For example, on WD15K, SCE improved Hit@1 by up to 5.5% and MRR by 4.2%.
  • Interpretability: SCE significantly improved interpretability metrics, achieving the highest Global Interpretability (GI) score (9.7) on the WD15K dataset, outperforming methods like NeuralLP and PS-Agent.
  • Efficiency: SCE demonstrated a substantial runtime advantage over PS-Agent, which relies on external rules. SCE incurred only a ~13% overhead compared to PS-Agent's ~1329% overhead on the WD15K dataset.
  • Ablation Studies: Removing either the entity or relation consistency module led to performance drops, confirming the importance of both components. SCE(w/o ent) still outperformed most baselines on denser datasets, highlighting the robustness of relation consistency.

Limitations and Future Work

  • Reliance on Pre-Trained Embeddings: SCE, like many path-based methods, depends on pre-trained embeddings. This might limit its adaptability to domains where such embeddings are unavailable or unsuitable.
  • Hyperparameter Tuning: The model has several important hyperparameters (e.g., weights for combining consistency scores) that require careful tuning, potentially increasing complexity.
  • Partial Solution to Spurious Paths: While SCE significantly reduces spurious paths, it doesn't entirely eliminate them. The authors suggest that more effective solutions are needed for comprehensive mitigation.

Future work could explore methods that are less reliant on pre-trained embeddings and develop more advanced techniques to fully address the spurious path problem.

Summary updated

BibTeX
@inproceedings{huang2025sce,
  author = {Yanwen Huang and Yao Liu and Qiao Liu and Rui Hou and Tingting Dai},
  title = {{SCE: Semantic Consistency Enhanced Reinforcement Learning for Multi-Hop Knowledge Graph Reasoning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure