Abstract
Knowledge graphs are used to represent relational information in terms of triples. To enable learning about domains, embedding models, such as tensor factorization models, can be used to make predictions of new triples. Often there is background taxonomic information (in terms of subclasses and subproperties) that should also be taken into account. We show that existing fully expressive (a.k.a. universal) models cannot provably respect subclass and subproperty information. We show that minimal modifications to an existing knowledge graph completion method enables injection of taxonomic information. Moreover, we prove that our model is fully expressive, assuming a lower-bound on the size of the embeddings. Experimental results on public knowledge graphs show that despite its simplicity our approach is surprisingly effective.
Introduction
Knowledge Graphs (KGs) such as Freebase and WordNet are foundational for representing relational data, yet they are notoriously incomplete. Link prediction, or KG completion, addresses this by learning entity and relation embeddings to predict missing facts. However, standard KGs are often accompanied by ontologies or taxonomies—sets of logical rules like "every President is a Person."
Existing "fully expressive" models, which can theoretically represent any set of facts, struggle to strictly adhere to these background rules. Because their embeddings can contain negative values, simple inequality constraints on vectors do not reliably translate to logical "if-then" guarantees. This paper introduces SimplE+, a modification to the SimplE model that bridges the gap between flexible machine learning and rigid symbolic logic.
Methodology
SimplE+ builds upon the SimplE tensor factorization approach, which assigns dual vectors to each entity (representing head and tail roles). The core innovations are:
- Non-negativity Constraints: Entity embeddings are restricted to non-negative values by passing them through a non-linear activation function, with ReLU proving most effective.
- Subsumption Constraints: To enforce that a relation $r$ is a subproperty of $s$ ($r \to s$), the model ensures the relation vector for $s$ is element-wise greater than or equal to the vector for $r$ ($s \ge r$).
With non-negative entity vectors, the mathematical inequality $s \ge r$ creates a monotonic relationship: the probability score for the broader relation $s$ will always be greater than or equal to the score for the specific relation $r$. Crucially, the authors prove that SimplE+ remains fully expressive, meaning it can represent any possible set of facts given sufficient dimensionality, despite these constraints.
Experimental Results
The researchers evaluated SimplE+ on standard benchmarks (WN18, FB15k) and specialized datasets with explicit taxonomic rules (Sport, Location).
Performance in Sparse Data
SimplE+ demonstrates significant advantages in data-sparse regimes. By incorporating background rules, the model acts as a powerful regularizer, focusing its learning capacity on non-obvious patterns rather than facts already implied by the taxonomy. On the Sport dataset, SimplE+ significantly outperforms both the base SimplE model and pure logical inference as training data decreases.
General Performance and Parity
A key concern when adding constraints is the potential for a "performance penalty" on general tasks. However, evaluations on WN18 and FB15k—where no explicit rules were applied—show that SimplE+ performs on par with the original SimplE and other state-of-the-art models like ComplEx. This confirms that the non-negativity constraint does not hinder the model's general representational power.
Conclusion
SimplE+ provides a simple yet mathematically rigorous method for incorporating taxonomic background information into knowledge graph embeddings. By enforcing non-negativity, it ensures logical consistency without sacrificing expressivity. Future research will focus on extending this framework to handle more complex logical structures, such as conjunctions and disjointness, and exploring methods to automatically induce taxonomies from raw data.
Summary updated