Probabilistic Relation Induction in Vector Space Embeddings

Zied Bouraoui, Shoaib Jameel, Steven SchockaertarXiv 2017 · OpenReview

Abstract

Word embeddings have been found to capture a surprisingly rich amount of syntactic and semantic knowledge. However, it is not yet sufficiently well-understood how the relational knowledge that is implicitly encoded in word embeddings can be extracted in a reliable way. In this paper, we propose two probabilistic models to address this issue. The first model is based on the common relations-as-translations view, but is cast in a probabilistic setting. Our second model is based on the much weaker assumption that there is a linear relationship between the vector representations of related words. Compared to existing approaches, our models lead to more accurate predictions, and they are more explicit about what can and cannot be extracted from the word embedding.

Introduction

The NLP community has long recognized that word embeddings, such as Word2Vec and GloVe, implicitly encode relational knowledge. A classic example is the vector arithmetic $v(\text{Paris}) - v(\text{France}) + v(\text{Japan}) \approx v(\text{Tokyo})$. However, using this property for relation induction—automatically identifying new word pairs to complete knowledge bases—presents significant challenges.

Standard translation-based methods often lack "type checking," leading to nonsensical but high-confidence predictions (e.g., applying a "capital of" vector to "dog"). Furthermore, relation induction typically suffers from a scarcity of training data, making standard classifiers like SVMs unstable. This paper introduces two probabilistic models designed to provide faithful confidence scores and robust performance even with limited training examples.

Methodology

The authors propose two distinct probabilistic approaches to score the likelihood that a word pair $(s, t)$ belongs to a specific relation.

1. The Translation Model

This model extends the traditional translation view by representing a relation as a multivariate Gaussian distribution over vector translations. Crucially, it incorporates "type checking" by modeling the valid source and target words as their own Gaussian distributions. To accept a pair, the model verifies both that the translation $t - s$ is probable for the relation and that $s$ and $t$ belong to the correct semantic types (e.g., countries and cities).

Conceptual illustration of the Translation Model, showing source and target word clusters (types) and the relational translations between them.

2. The Regression Model

The translation assumption requires source and target vector spaces to be perfectly aligned, which often fails for looser semantic relations like meronyms (part-whole). The Regression Model relaxes this, assuming only a linear mapping from the source to the target space. Using Bayesian linear regression, it predicts the target word from the source. To handle high-dimensional embeddings with few training examples, the authors employ Singular Value Decomposition (SVD) for low-rank approximation.

Evaluation and Results

The models were evaluated using Skip-gram and GloVe embeddings across three benchmark datasets: the Google Analogy Test Set, the Bigger Analogy Test Set (BATS), and the DiffVec (DV) set.

Quantitative comparison of the proposed Translation (Trans) and Regression (Regr) models against baselines (3CA, LRC, SVM) across various embeddings and datasets.

As shown in the results table, the Translation model consistently outperformed strong baselines like 3CosAvg and SVMs. For instance, on Google News Skip-gram embeddings, it achieved a Mean Average Precision (MAP) of 0.890 on the Google dataset, significantly higher than the SVM's 0.611.

While the Translation model was the overall winner, the Regression model excelled in specific relations where the strict translation assumption was too rigid, such as Event or Hypernym relations in the DiffVec set, provided sufficient training data was available to learn the linear mapping.

Key Insights

  • Type-checking is essential: Explicitly modeling the domain and range of a relation drastically reduces false positives.
  • Distributions vs. Vectors: Modeling relations as probability distributions allows the system to ignore irrelevant semantic features and weight important ones appropriately.
  • Geometric Flexibility: While translation is a strong geometric assumption, many relationships are better described by linear transformations, highlighting the need for the regression-based approach in complex semantic tasks.

Conclusion

By moving from deterministic vector math to probabilistic modeling, this research provides a more reliable framework for extracting relational knowledge from embeddings. Future work may involve combining these models or exploring vector concatenations to capture even deeper interactions between source and target concepts.

Summary updated

Abstract

Word embeddings have been found to capture a surprisingly rich amount of syntactic and semantic knowledge. However, it is not yet sufficiently well-understood how the relational knowledge that is implicitly encoded in word embeddings can be extracted in a reliable way. In this paper, we propose two probabilistic models to address this issue. The first model is based on the common relations-as-translations view, but is cast in a probabilistic setting. Our second model is based on the much weaker assumption that there is a linear relationship between the vector representations of related words. Compared to existing approaches, our models lead to more accurate predictions, and they are more explicit about what can and cannot be extracted from the word embedding.
BibTeX
@misc{bouraoui2017probabilistic,
  author = {Zied Bouraoui and Shoaib Jameel and Steven Schockaert},
  title = {{Probabilistic Relation Induction in Vector Space Embeddings}},
  year = {2017},
  eprint = {1708.06266},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1708.06266},
}
Figure