Abstract: Latent factor models are increasingly popular for modeling multi-relational knowledge graphs. By their vectorial nature, it is not only hard to interpret why this class of models works so well, but also to understand where they fail and how they might be improved. We conduct an experimental survey of state-of-the-art models, not towards a purely comparative end, but as a means to get insight about their inductive abilities. To assess the strengths and weaknesses of each model, we create simple tasks that exhibit first, atomic properties of binary relations, and then, common inter-relational inference through synthetic genealogies. Based on these experimental results, we propose new research directions to improve on existing models.
Introduction
In the realm of statistical relational learning and knowledge graph (KG) completion, latent factor models (LFMs) have largely replaced older, first-order logic methods. LFMs embed entities and relations into continuous vector spaces, offering superior predictive performance and scalability. However, because LFMs do not use explicit symbolic rules, they often function as "black boxes." This research investigates whether these vector-based models can actually learn to reason inductively from observed facts by stripping away the noise of real-world datasets to isolate their pure reasoning capabilities.
Models and Evaluation Framework
The authors evaluate six prominent LFMs to understand their inductive strengths and weaknesses. These models represent different approaches to defining scoring functions that measure the likelihood of a triple $(s, r, o)$ being true.
Instead of benchmarking on standard, noisy datasets like FB15k, the study utilizes deterministic, noise-free synthetic experiments. This approach isolates the models' inductive abilities from their capacity to handle uncertainty. The models are tested on two primary tasks:
- Learning Relation Properties: Evaluating the ability to learn reflexivity, symmetry, transitivity, and antisymmetry.
- Learning Inter-Relational Patterns: Testing if models can deduce complex kinship relations (e.g., uncle, cousin) from base relations (e.g., mother, father).
Experimental Setup
Two new synthetic benchmarks were designed for these tasks:
- Synthetic Relation Properties: Randomly generated $50 \times 50$ matrices exhibiting 13 valid combinations of set-theory properties.
- Synthetic Family Trees: Five independent, completely disjoint family trees spanning three generations, containing 115 entities and 17 kinship relations.
To test generalization, a "Family split" strategy was employed. Models were trained on all relations for four families but only base relations for a fifth family. The model must then predict complex relations for the fifth family, testing its ability to transfer logical rules to a disconnected subgraph.
Key Insights and Results
Transitivity vs. Antisymmetry
Most models easily learn transitivity because it introduces significant structural regularity. However, models like DistMult and CP fail completely on antisymmetric relations. ComplEx, utilizing complex conjugates, handles both exceptionally well, achieving near-perfect Average Precision (AP $\approx 1.0$) across various property combinations.
The Disjoint Subgraph Failure
The most significant finding is the failure of LFMs in the "Family split" experiment. Even if a model learns the rule for "grandfather" perfectly in four families, it cannot apply that rule to a fifth, completely disconnected family. When zero complex relations were observed for the target family during training, all models failed, with AP dropping to baseline levels.
Why Models Fail to Transfer
LFMs rely on the dot products of specific entity embeddings. If two subgraphs never interact, their embeddings share no constraints other than the global relation vector. The mathematical framework does not force disconnected entities to align in a way that satisfies a universal logical rule; instead, the models learn local geometric alignments specific to the entities seen during training.
Conclusion and Future Directions
While LFMs are powerful for interpolating within connected graphs, they struggle with true inductive generalization to disjoint sets of entities. To address these limitations, the authors suggest:
- Architectural Blending: Combining scoring functions (e.g., ComplEx's embeddings with TransE's bilinear terms) to balance expressiveness and robustness.
- Solving the Transfer Problem: Exploring parameter sharing across entity representations or integrating explicit logic rules into the training process to force models to learn universal rules rather than local geometric tricks.
Summary updated