Neural Probabilistic Logic Learning for Knowledge Graph Reasoning

Fengsong Sun, Jinyu Wang, Zhiqing Wei, Xianchao Zhang

TL;DR

Neural Probabilistic Logic Learning (NPLL) is a new framework that bridges the gap between precise-but-stiff logical rules and flexible-but-opaque neural embeddings for knowledge graph reasoning. By treating a neural network as an inference engine for a Markov Logic Network, NPLL achieves state-of-the-art accuracy with significantly fewer parameters and excels in "zero-shot" scenarios where data is scarce.

What Problem Does This Paper Address?

Knowledge Graph (KG) reasoning—the task of predicting missing links between entities (e.g., predicting that Paris is the capital of France)—has traditionally been split into two camps:

  1. Rule-based methods: These use symbolic logic (e.g., "If A is the father of B, and B is the father of C, then A is the grandfather of C"). They are highly interpretable and precise but struggle to scale to massive graphs and can't handle the "fuzzy" nature of real-world data.
  2. Embedding-based methods: These map entities and relations into high-dimensional vector spaces. They are incredibly scalable and catch subtle patterns, but they are "black boxes" that often struggle with complex logical relationships and require massive amounts of training data.

The authors address the "accuracy vs. scalability" trade-off, seeking a method that retains the interpretability of logic while leveraging the power of deep learning.

Methods

The authors propose Neural Probabilistic Logic Learning (NPLL). The core idea is to integrate Markov Logic Networks (MLNs)—a framework that attaches weights to logical rules—with a neural Scoring Module.

Key components include:

  • Variational Inference & EM Algorithm: The model uses an Expectation-Maximization (EM) approach. In the E-step, a neural network (the Scoring Module) estimates the probability of unknown facts. In the M-step, the weights of the logical rules are updated to better fit the data.
  • The Scoring Module: This is the "neural" heart of the system. It uses a bilinear tensor product and linear layers to calculate a compatibility score for any given triplet (head entity, relation, tail entity). This score acts as an approximate posterior probability for the logic model.
  • Rule Integration: Unlike pure embedding models, NPLL incorporates prior knowledge in the form of first-order logic rules (mined using tools like NeuralLP). The model learns which rules are most reliable by adjusting their weights ($\omega$) during training.

Datasets

The model was evaluated on four standard benchmarks that vary in size and domain:

  • UMLS: A small, domain-specific medical dataset.
  • Kinship: A dataset of family relationships.
  • FB15k-237: A challenging, pruned version of Freebase containing general world knowledge.
  • WN18RR: A subset of WordNet focused on lexical relations between words.

To test robustness, the authors also created subsets of FB15k-237 with varying levels of data sparsity (from 0% to 20% training data) and a specific zero-shot dataset where the relations in the test set never appeared in the training set.

Key Insights and Ideas

  • The "Neural Inference" Shift: Instead of using neural networks just to learn embeddings, NPLL uses them to perform the heavy lifting of probabilistic inference for a logic model. This makes the logic model much more efficient on large graphs.
  • Efficiency is King: NPLL-basic achieves better results than previous heavyweights like ExpressGNN while using only one-fourth of the parameters (~65M vs ~251M).
  • Logic as a Safety Net: Because the model relies on logical rules, it doesn't "break" when it encounters a relation it hasn't seen many times. Pure embedding methods fail in these "long-tail" or zero-shot scenarios because they have no vector representation to rely on; NPLL simply falls back on the logic.
  • Data Efficiency: The results on the "FB-0" dataset (where training data is nearly non-existent) show that NPLL can reason effectively using almost entirely the structure of the rules and the graph, whereas other models' performance collapses.

Key Results

  • State-of-the-Art Accuracy: NPLL-basic outperformed all competitors (including TransE, RotatE, and NeuralLP) across almost all metrics. For example, on WN18RR, it achieved a Mean Reciprocal Rank (MRR) of 0.7668, compared to the next best rule-learning method at 0.6700.
  • Zero-Shot Dominance: In zero-shot tests on FB15k-237, embedding-based methods like RotatE dropped to an MRR of near zero (0.006). NPLL-basic maintained a robust MRR of 0.4425.
  • Interpretability: By looking at the learned rule weights, researchers can see exactly which logical "shortcuts" the model is using to reach its conclusions.

Limitations and Future Work

  • Rule Mining Dependency: NPLL isn't entirely "end-to-end" in the sense that it requires a pre-existing set of candidate logical rules. If the rule-mining step (using NeuralLP) misses a key logical connection, NPLL might not be able to recover it.
  • Hyperparameter Tuning: The performance is sensitive to the confidence threshold ($\alpha$) used to filter rules. The authors had to perform extensive experiments to find the optimal threshold for each dataset.
  • Future Directions: The authors suggest that further tuning the "encoding network" (the part that extracts features from the graph) could lead to even higher performance boosts.

Summary updated

Abstract

Knowledge graph (KG) reasoning is a task that aims to predict unknown facts based on known factual samples. Reasoning methods can be divided into two categories: rule-based methods and KG-embedding based methods. The former possesses precise reasoning capabilities but finds it challenging to reason efficiently over large-scale knowledge graphs. While gaining the ability to reason over large-scale knowledge graphs, the latter sacrifices reasoning accuracy. This paper aims to design a reasoning framework called Neural Probabilistic Logic Learning(NPLL) that achieves accurate reasoning on knowledge graphs. Our approach introduces a scoring module that effectively enhances the expressive power of embedding networks, striking a balance between model simplicity and reasoning capabilities. We improve the interpretability of the model by incorporating a Markov Logic Network based on variational inference. We empirically evaluate our approach on several benchmark datasets, and the experimental results validate that our method substantially enhances the accuracy and quality of the reasoning results.
BibTeX
@misc{sun2024neural,
  author = {Fengsong Sun and Jinyu Wang and Zhiqing Wei and Xianchao Zhang},
  title = {{Neural Probabilistic Logic Learning for Knowledge Graph Reasoning}},
  year = {2024},
  eprint = {2407.03704},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.03704},
}
Figure