On Inductive Abilities of Latent Factor Models for Relational Learning

Théo Trouillon, Éric Gaussier, Christopher R. Dance, Guillaume Bouchard

Latent Factor Models (LFMs) have become the industry standard for predicting missing links in Knowledge Graphs (KGs) due to their impressive scalability. However, these models are often treated as "black boxes." This research, published in the Journal of Artificial Intelligence Research (2019), subjects these models to a diagnostic "stress test" to uncover their inductive reasoning capabilities and identify exactly where their reasoning breaks down.

The Challenge of Relational Reasoning

Knowledge Graphs—networks of entities and their relationships—are almost always incomplete. To fill these gaps, LFMs represent entities and relations as high-dimensional vectors (embeddings). While these models perform well on large-scale benchmarks like Freebase, their success is largely statistical. Unlike symbolic logic, LFMs do not follow explicit rules. The authors aim to move beyond simple performance leaderboards to understand the "inductive DNA" of these models—specifically, whether they can learn fundamental logical properties like symmetry, transitivity, and inter-relational inference.

Methods and Model Architectures

The study evaluates six prominent LFMs: CP, RESCAL, TransE, F model, DistMult, and ComplEx. These models differ in how they define the scoring function for a triple and the space in which their parameters reside.

Summary of the Latent Factor Models evaluated, including their scoring functions and parameter spaces.

The researchers moved away from noisy real-world data to a "clean room" environment using synthetic datasets. This allowed for the isolation of specific logical challenges:

  1. Atomic Properties: Can the model learn that a relation is symmetric (friendship), antisymmetric (older than), or transitive (located in)?
  2. Inter-relational Inference: Can the model learn compositional rules, such as "a mother’s mother is a grandmother"?
  3. Transfer Learning: If a model learns the rules of a family tree for one group, can it apply those same rules to a completely separate, disjoint group of entities?

Experimental Setup and Datasets

The benchmarking utilized three primary data structures:

  • Property Matrices: $50 \times 50$ matrices specifically engineered to exhibit combinations of reflexivity, symmetry, and transitivity.
  • Synthetic Genealogies: A kinship dataset involving five independent families (115 entities total) with 17 relations (mother, uncle, cousin, etc.).
  • The "Family Split": A novel benchmarking approach where the model is trained on four families and tested on a fifth, disjoint family. This tests if a model can abstract the "concept" of a relation independently of the specific people it saw during training.

Key Insights: Strengths and Failures

The Bias of Symmetry

A significant finding was the mathematical bias inherent in certain architectures. For instance, DistMult is inherently symmetric; it cannot distinguish between "A is the father of B" and "B is the father of A." Only ComplEx (utilizing complex numbers) and RESCAL (utilizing matrices) successfully mastered antisymmetry.

The Ease of Transitivity

Interestingly, almost all tensor factorization models proved naturally adept at learning transitivity. They could reconstruct transitive relations perfectly even with very low-dimensional embeddings, suggesting that the algebraic structure of these models aligns well with transitive properties.

The "Disjoint Entity" Wall

The most critical discovery was the failure of LFMs in the "Family Split" scenario. Because LFMs rely heavily on specific entity embeddings, their performance plummeted when asked to predict relations in a new family. The models failed to "abstract" the rules of kinship; they only knew how those rules applied to the specific people they saw during training. In tests with zero overlap between families, all models failed to reach an Average Precision (AP) above 0.4, whereas a logic-based system would achieve a perfect 1.0.

Conclusion and Future Directions

The research highlights a fundamental limitation in current LFMs: the absence of parameter sharing. Since each entity has its own unique vector, there is no mechanism to share the "logic" of a relation across different parts of a graph.

ComplEx emerged as the most balanced model, handling symmetry, antisymmetry, and transitivity near-perfectly while remaining computationally efficient. To move forward, the authors suggest exploring hybrid models that combine geometric and algebraic strengths, or finding ways to force models to obey first-order logic rules through regularization.

Summary updated

Abstract

Latent factor models are increasingly popular for modeling multi-relational knowledge graphs. By their vectorial nature, it is not only hard to interpret why this class of models works so well, but also to understand where they fail and how they might be improved. We conduct an experimental survey of state-of-the-art models, not towards a purely comparative end, but as a means to get insight about their inductive abilities. To assess the strengths and weaknesses of each model, we create simple tasks that exhibit first, atomic properties of binary relations, and then, common inter-relational inference through synthetic genealogies. Based on these experimental results, we propose new research directions to improve on existing models.
BibTeX
@article{trouillon2019inductive,
  author = {Théo Trouillon and Éric Gaussier and Christopher R. Dance and Guillaume Bouchard},
  title = {{On Inductive Abilities of Latent Factor Models for Relational Learning}},
  journal = {J. Artif. Intell. Res.},
  year = {2019},
  doi = {10.1613/jair.1.11305},
}
Figure