Combining Embeddings and Rules for Fact Prediction (Invited Paper)

Armand Boschin, Nitisha Jain, Gurami Keretchashvili, Fabian M. Suchanek

TL;DR

This paper surveys recent advancements in combining symbolic rule mining with neural knowledge graph embeddings for more accurate fact prediction in incomplete knowledge bases. It highlights how these complementary approaches can overcome individual limitations, leading to more robust and interpretable AI systems.

What Problem Does This Paper Address?

Knowledge bases (KBs), like Wikidata or DBpedia, are vast repositories of information about the world, but they are inherently incomplete. This means many facts that are true in reality are missing from these KBs. The challenge is to predict these missing facts to make KBs more useful for applications like question answering and search. Two primary methods have emerged: Rule Mining, which discovers symbolic rules (e.g., "if two people are married, they likely live in the same city"), and Knowledge Graph Embeddings (KGEs), which represent entities and relations as vectors in a continuous space. While rule mining offers interpretability and can handle logical constraints, it can be brittle and struggle with exceptions. KGEs excel at capturing complex patterns and handling large-scale data but often lack interpretability and can be sensitive to data quality. This paper addresses the need to bridge this gap by exploring how combining these two approaches can lead to better fact prediction.

Methods

The paper surveys various methods that combine rule mining and KGEs, broadly categorizing them into three classes:

  • Adding Simple Axioms: These approaches incorporate basic logical constraints (like inverse, symmetric, or equivalent relations) directly into the KGE training process. Examples include enforcing that if (h, r, t) is true, then (t, r_inverse, h) should also be true, by modifying the scoring functions or embedding spaces.
  • Complex Constraints: This category extends the previous one by supporting more general and complex logical constraints derived from ontologies. Methods here might use logical reasoners to identify inconsistencies and then use these as negative samples for retraining KGE models.
  • Joint Learning: These approaches learn embeddings and rule confidences simultaneously. This often involves defining a joint loss function that balances predicting known facts with imitating the confidence scores of learned rules.

The paper also details various KGE models (TransE, TransH, TransR, RotatE, HAKE, RESCAL, DistMult, Complex, SimplE, ConvE, ConvKB, GCNs, R-GCNs, VR-GCN, SACN, CompGCN) and rule mining techniques (AMIE, RuDiK, AnyBURL, DRUM).

Datasets

The paper mentions several standard datasets used for evaluating KGEs:

  • FB15k: Extracted from Freebase.
  • WN18: A subset of WordNet.
  • Yago3-10: A subset of Yago3 focusing on people.

It also notes the creation of modified versions of these datasets to address issues like data leakage:

  • FB15k-237: FB15k with inverse relations removed.
  • WN18RR: WN18 with issues addressed.
  • Yago3-TC: A dataset for evaluating KG completion, including true and false facts.

Key Insights and Ideas

  • Complementarity is Key: Rule mining and KGEs have distinct strengths and weaknesses. Combining them offers a synergistic approach where one compensates for the other's limitations, leading to more robust and interpretable predictions.
  • Interpretability vs. Scalability: Rule mining provides human-understandable rules, crucial for explaining predictions. KGEs offer scalability and the ability to capture subtle patterns but often lack transparency. Neuro-symbolic methods aim to get the best of both worlds.
  • Beyond Simple Rules: While early work focused on simple axioms, more recent research explores complex ontological reasoning and joint learning of rules and embeddings to capture richer semantic relationships.
  • Data Quality Matters: The paper highlights significant issues with benchmark datasets, such as data leakage (inverse relations, duplicate relations) and sparsity, which can artificially inflate model performance and mask true capabilities. This underscores the need for careful evaluation and new benchmarks.
  • Negative Examples are Crucial: Both rule mining and KGEs benefit from negative examples (facts known to be false) to avoid over-generalization. Generating these effectively, especially when KBs lack explicit negative assertions, is a significant research area.

Key Results

The paper is a survey, so it doesn't present new experimental results but rather synthesizes findings from existing research. Key findings highlighted include:

  • Neuro-symbolic approaches that combine embeddings with logical constraints (axioms, rules) generally show improved performance on link prediction tasks compared to using either method alone.
  • Methods incorporating taxonomic information (like HAKE) or ontological reasoning (like ReasonKGE) demonstrate better performance by leveraging background knowledge.
  • Studies on benchmark datasets reveal that many popular KGE models perform artificially well due to data leakage and redundancy, with performance dropping significantly when these issues are addressed.
  • Some research suggests that simple rule-based techniques can outperform complex embedding techniques on realistic, cleaned datasets.

Limitations and Future Work

  • Limitations of Existing Methods:
    • Rule mining can be restricted to relations appearing in rules and may struggle with exceptions or low confidence.
  • KGEs often project away literals and numerical values, limiting their ability to handle certain types of information.
  • Benchmark datasets suffer from data leakage and redundancy, making evaluation challenging.
  • Solving integer linear programming problems for constraints can be time-consuming, limiting scalability.
  • Future Work Directions:
    • The paper suggests that using embeddings to improve rule mining is a less explored but promising direction.
  • Developing better evaluation protocols and benchmarks that reflect realistic scenarios is crucial.
  • Further research into methods that can handle many-to-many relations and improve scalability for complex constraint satisfaction is needed.
  • Exploring more sophisticated ways to integrate symbolic knowledge (like axioms and rules) into embedding models to enhance interpretability and reasoning capabilities.

Summary updated

Abstract

Knowledge bases are typically incomplete, meaning that they are missing information that we would expect to be there. Recent years have seen two main approaches to guess missing facts: Rule Mining and Knowledge Graph Embeddings. The first approach is symbolic, and finds rules such as "If two people are married, they most likely live in the same city". These rules can then be used to predict missing statements. Knowledge Graph Embeddings, on the other hand, are trained to predict missing facts for a knowledge base by mapping entities to a vector space. Each of these approaches has their strengths and weaknesses, and this article provides a survey of neuro-symbolic works that combine embeddings and rule mining approaches for fact prediction.
BibTeX
@inproceedings{boschin2022combining,
  author = {Armand Boschin and Nitisha Jain and Gurami Keretchashvili and Fabian M. Suchanek},
  title = {{Combining Embeddings and Rules for Fact Prediction (Invited Paper)}},
  booktitle = {AIB},
  year = {2022},
  url = {https://openreview.net/forum?id=75tyXY5m-gV},
}
Figure