Improved Knowledge Graph Embedding Using Background Taxonomic Information

Bahare Fatemi, Siamak Ravanbakhsh, David Poole

Abstract

Knowledge graphs are used to represent relational information in terms of triples. To enable learning about domains, embedding models, such as tensor factorization models, can be used to make predictions of new triples. Often there is background taxonomic information (in terms of subclasses and subproperties) that should also be taken into account. We show that existing fully expressive (a.k.a. universal) models cannot provably respect subclass and subproperty information. We show that minimal modifications to an existing knowledge graph completion method enables injection of taxonomic information. Moreover, we prove that our model is fully expressive, assuming a lower-bound on the size of the embeddings. Experimental results on public knowledge graphs show that despite its simplicity our approach is surprisingly effective.

Introduction

Knowledge Graphs (KGs) like Freebase or WordNet store massive networks of facts as triples: (head, relation, tail). Because these graphs are inherently incomplete, embedding models are used to map entities and relations into high-dimensional vector spaces to predict missing links. However, KGs often include taxonomies—background logic defining hierarchies (e.g., "every President is a Person").

Standard "fully expressive" embedding models like ComplEx and SimplE struggle to guarantee these logical rules. If a model predicts an entity is a "President" but not a "Person," it fails a basic consistency check. Mathematically, these models cannot enforce such rules for all possible entities due to their handling of negative values. This paper introduces SimplE+, a modified model that incorporates non-negative entity vectors to bake taxonomic rules directly into the underlying mathematics.

Methods: The SimplE+ Model

The authors propose SimplE+, a variation of the state-of-the-art SimplE model, with several key modifications:

  1. Non-Negativity Constraint: Entity embeddings are restricted to be non-negative using a ReLU or similar non-linearity.
  2. Monotonicity: In dot-product-based models, non-negative entity vectors ensure that increasing relation vector values monotonically increases the overall score.
  3. Enforcing Subsumption: To represent a rule like "Relation $r$ implies Relation $s$", the relation vector for $r$ is defined as $s$ minus a non-negative "difference" vector $\delta_r$.
  4. Expressivity: The authors provide a mathematical proof that SimplE+ remains "fully expressive," meaning it can represent any configuration of facts given sufficient dimensions.

Experimental Evaluation

The model was evaluated on standard large-scale benchmarks (WN18 and FB15k) and specialized datasets (Sport and Location) containing explicit taxonomic rules.

Key Results

  • Performance Stability: On standard datasets without explicit taxonomy, SimplE+ performed on par with the original SimplE, proving that non-negativity constraints do not hinder the model's general learning capability.
  • Taxonomic Gains: On the Sport and Location datasets, SimplE+ outperformed standard models significantly. In scenarios where redundant triples were removed to force reliance on logic, SimplE+ showed a 90% improvement in Hit@1 for Sport and a 230% improvement for Location.
Performance comparison (HIT@1) across varying proportions of training data.
  • Efficiency and Convergence: SimplE+ reached optimal performance in significantly fewer training iterations than the standard SimplE model, demonstrating superior learning efficiency.
Convergence speed comparison showing Loss Value over iterations.

Conclusion

SimplE+ demonstrates that baking background logic into the architecture of a knowledge graph embedding model acts as a powerful regularizer. By ensuring logical consistency through non-negativity and monotonicity, the model achieves faster convergence and superior performance in data-sparse environments. Future work will explore extending this framework to handle more complex logical structures beyond simple hierarchical subsumption.

Summary updated

Abstract

Knowledge graphs are used to represent relational information in terms of triples. To enable learning about domains, embedding models, such as tensor factorization models, can be used to make predictions of new triples. Often there is background taxonomic information (in terms of subclasses and subproperties) that should also be taken into account. We show that existing fully expressive (a.k.a. universal) models cannot provably respect subclass and subproperty information. We show that minimal modifications to an existing knowledge graph completion method enables injection of taxonomic information. Moreover, we prove that our model is fully expressive, assuming a lower-bound on the size of the embeddings. Experimental results on public knowledge graphs show that despite its simplicity our approach is surprisingly effective.
BibTeX
@inproceedings{fatemi2019improved,
  author = {Bahare Fatemi and Siamak Ravanbakhsh and David Poole},
  title = {{Improved Knowledge Graph Embedding Using Background Taxonomic Information}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=1Cv_PrVI7Pj},
}
Figure