Context Vectors are Reflections of Word Vectors in Half the Dimensions

Zhenisbek Assylbekov, Rustem TakhanovJAIR 2019 · OpenReview

TL;DR

This paper offers a theoretical explanation for the relationship between word and context embeddings in models like word2vec. It reveals that context vectors are essentially reflections of word vectors across roughly half their dimensions, a finding that has direct implications for improving the efficiency of training these models.

What Problem Does This Paper Address?

Word embeddings, like those produced by word2vec, are fundamental to many Natural Language Processing (NLP) tasks. However, a deep theoretical understanding of how these embeddings are formed and relate to each other, particularly the distinction between "word" (input) embeddings and "context" (output) embeddings, has been lacking. This paper aims to bridge this gap by providing a theoretical analysis of the relationship between these two types of embeddings, specifically within the Skip-gram Negative Sampling (SGNS) model. Understanding this relationship is crucial for developing better embeddings and more interpretable NLP models.

Methods

The authors build their argument on three core probabilistic assumptions about word vectors, context vectors, and text generation. These assumptions are grounded in existing literature and empirical observations.

  1. Isotropic Gaussian Word Vectors: They assume that initial word vectors are drawn independently and identically from an isotropic multivariate Gaussian distribution. This is a common starting point in theoretical analyses of word embeddings.
  2. Orthogonal Transformation for Context Vectors: A key assumption is that context vectors are derived from word vectors through an orthogonal transformation. This preserves inner products and norms, reflecting empirical observations that word and context embeddings share similar geometric properties.
  3. Context Probability Model: They adopt a model where the probability of a word appearing in a context is proportional to the product of its unigram probability and an exponential term involving the dot product of the word and context vectors. This model is similar to existing log-linear models but with specific choices for capturing word frequencies.

Using these assumptions, they mathematically demonstrate that the word-word Pointwise Mutual Information (PMI) matrix is approximately a random symmetric Gaussian ensemble. This leads to their main theoretical result: context vectors are reflections of word vectors in approximately half the dimensions.

Datasets

The paper uses standard datasets for empirical verification:

  • text8: A widely used dataset derived from Wikipedia.
  • enwik9: Another large Wikipedia corpus.

These datasets are used to extract PMI matrices for empirical analysis, and the resulting word embeddings (both standard SGNS and their proposed weight-tied SGNS+WT) are evaluated on standard word similarity and analogy tasks.

Key Insights and Ideas

  • Context Vectors as Reflections: The most significant insight is the theoretical derivation that context vectors are reflections of word vectors in approximately half the dimensions. This provides a concrete geometric interpretation of their relationship.
  • PMI Matrix as a Gaussian Ensemble: The paper shows that the word-word PMI matrix, a key component in understanding word relationships, behaves like a random symmetric Gaussian matrix under their assumptions. This connection to random matrix theory is powerful for theoretical analysis.
  • Q is Approximately Involutary: Their analysis suggests that the orthogonal transformation matrix Q, which maps word vectors to context vectors, is approximately an "involutary" matrix. This means Q² is close to the identity matrix, implying that applying the transformation twice is almost equivalent to doing nothing.
  • Theoretical Justification for Weight Tying: The findings provide a strong theoretical basis for tying word and context embeddings in SGNS models. This is achieved by setting the transformation matrix Q to a specific diagonal form where elements are ±1, effectively "flipping" some dimensions.

Key Results

  • Theorem 1: Context vectors are reflections of word vectors in approximately half the dimensions.
  • Empirical Validation: The paper empirically validates their theoretical findings by showing that the eigenvalues of real-world PMI matrices exhibit a symmetric distribution around zero, consistent with their Gaussian ensemble model.
  • SGNS+WT Performance: They propose a weight-tied SGNS model (SGNS+WT) based on their theoretical insights. This model achieves comparable performance to the standard SGNS model on word similarity and analogy tasks, despite using approximately 50% fewer parameters. This demonstrates the practical utility of their theoretical work.

Limitations and Future Work

  • Approximations: The theoretical results rely on approximations (e.g., PMI matrix being a Gaussian ensemble, Q being involutary). While these approximations hold well for typical high-dimensional word embeddings, their exactness is not guaranteed.
  • Hypothesis on Semantics vs. Syntax: The authors hypothesize that the "flipped" dimensions (x embeddings) are more related to semantics, while the "unflipped" dimensions (y embeddings) are more related to syntax. Testing this hypothesis is deferred to future work.
  • Further Exploration of Q: While they propose a specific diagonal form for Q, the paper notes that any involutary matrix can be represented in a similar form, suggesting further avenues for exploring the structure of Q.
  • Application to Other Models: The theoretical framework could potentially be extended to analyze other word embedding models beyond SGNS.

Summary updated

Abstract

This paper takes a step towards theoretical analysis of the relationship between word embeddings and context embeddings in models such as word2vec. We start from basic probabilistic assumptions on the nature of word vectors, context vectors, and text generation. These assumptions are supported either empirically or theoretically by the existing literature. Next, we show that under these assumptions the widely-used word-word PMI matrix is approximately a random symmetric Gaussian ensemble. This, in turn, implies that context vectors are reflections of word vectors in approximately half the dimensions. As a direct application of our result, we suggest a theoretically grounded way of tying weights in the SGNS model.
BibTeX
@article{assylbekov2019context,
  author = {Zhenisbek Assylbekov and Rustem Takhanov},
  title = {{Context Vectors are Reflections of Word Vectors in Half the Dimensions}},
  journal = {J. Artif. Intell. Res.},
  year = {2019},
  url = {https://openreview.net/forum?id=nEsK-LnzsDa},
}
Figure