Injecting Logical Background Knowledge into Embeddings for Relation Extraction

Tim Rocktäschel, Sameer Singh, Sebastian Riedel

Introduction

In the mid-2010s, relation extraction (RE)—the task of identifying relationships between entities in text—was caught between two paradigms. Matrix Factorization (MF) excelled at uncovering latent patterns in large datasets but struggled with the "cold start" problem in sparse data scenarios. Conversely, rule-based systems provided precision and interpretability but lacked the ability to generalize. This paper by Rocktäschel, Singh, and Riedel (2015) proposes an elegant middle ground: injecting logical rules directly into the low-dimensional vector space of embeddings.

The Challenge of Universal Schema

The authors focus on the "Universal Schema" approach, where relation extraction is modeled as a matrix completion problem. The matrix rows represent entity pairs (e.g., "Obama, USA") and columns represent relations, including both structured database relations (e.g., born_in) and surface-form textual patterns. Standard MF fails when a relation has no observed facts in the training set (zero-shot). While humans can provide simple rules like "If X is the CEO of Y, then X works for Y," the challenge lies in forcing a continuous vector space to adhere to these discrete logical constraints.

Methodology: Joint Optimization with Logic

The core innovation is a method to jointly train entity-pair and relation embeddings using both factual data and first-order logic rules.

Overview of the proposed framework: injecting first-order logic into low-rank embeddings.

Rather than using logic as a pre-processing step, the authors treat logical formulas as differentiable objectives. Using Product T-Norms, they translate logical implications into math: the probability of an implication $A \Rightarrow B$ is calculated as $P(A) \cdot P(B) - P(A) + 1$. Because the probability of a fact is modeled as a sigmoid of the dot product of embeddings ($\sigma(v_r \cdot v_p)$), the entire logical formula becomes differentiable. During training, the model uses Stochastic Gradient Descent (SGD) to minimize a loss function that combines factual loss (errors in predicting known facts) and logical loss (violations of the provided rules).

Experimental Results

The model was evaluated on the NY Times Corpus and Freebase, using 36 high-quality mined rules (e.g., #2-executive-#1(x, y) \Rightarrow person/company(x, y)).

Performance comparison across various relations, showing the superiority of the Joint model in MAP and weighted MAP.

Zero-Shot Success and Data Efficiency

In a "zero-shot" test where training alignments for certain relations were removed, the Joint Optimization model achieved a Mean Average Precision (MAP) of 0.52, compared to effectively 0.01 for standard MF. Furthermore, the model demonstrated significant data efficiency; with only 10% of training data, the Joint model reached a performance level (wMAP ~0.45) that standard MF required far more data to achieve.

Generalization Beyond Rules

A key insight is that logic acts as a regularizer. Because the model learns embeddings, a rule about "CEO of" doesn't just help that specific relation; it benefits any textual pattern with a similar embedding, even if it wasn't in the original rule. This allows the model to capture asymmetric relationships (where $A \Rightarrow B$ but $B \not\Rightarrow A$) more accurately than pure MF.

Conclusion

This work demonstrates that integrating symbolic logic into latent factor models provides a powerful way to handle data sparsity and incorporate human knowledge. By treating logic as a guide during the learning process, the authors achieve state-of-the-art performance in relation extraction, particularly in challenging zero-shot and low-data regimes.

Summary updated

Abstract

© 2015 Association for Computational Linguistics.Matrix factorization approaches to relation extraction provide several attractive features: they support distant supervision, handle open schemas, and leverage unlabeled data. Unfortunately, these methods share a shortcoming with all other distantly supervised approaches: they cannot learn to extract target relations without existing data in the knowledge base, and likewise, these models are inaccurate for relations with sparse data. Rule-based extractors, on the other hand, can be easily extended to novel relations and improved for existing but inaccurate relations, through first-order formulae that capture auxiliary domain knowledge. However, usually a large set of such formulae is necessary to achieve generalization. In this paper, we introduce a paradigm for learning low-dimensional embeddings of entity-pairs and relations that combine the advantages of matrix factorization with first-order logic domain knowledge. We introduce simple approaches for estimating such embeddings, as well as a novel training algorithm to jointly optimize over factual and first-order logic information. Our results show that this method is able to learn accurate extractors with little or no distant supervision alignments, while at the same time generalizing to textual patterns that do not appear in the formulae.
BibTeX
@inproceedings{rocktaschel2015injecting,
  author = {Tim Rocktäschel and Sameer Singh and Sebastian Riedel},
  title = {{Injecting Logical Background Knowledge into Embeddings for Relation Extraction}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
  doi = {10.3115/v1/n15-1118},
}
Figure