Word2vec Conjecture and A Limitative Result

Falcon Z. DaiarXiv 2020 · OpenReview

Abstract

Being inspired by the success of \texttt{word2vec} \citep{mikolov2013distributed} in capturing analogies, we study the conjecture that analogical relations can be represented by vector spaces. Unlike many previous works that focus on the distributional semantic aspect of \texttt{word2vec}, we study the purely \emph{representational} question: can \emph{all} semantic word-word relations be represented by differences (or directions) of vectors? We call this the word2vec conjecture and point out some of its desirable implications. However, we will exhibit a class of relations that cannot be represented in this way, thus falsifying the conjecture and establishing a limitative result for the representability of semantic relations by vector spaces over fields of characteristic 0, e.g., real or complex numbers.

Introduction

The success of word2vec in solving linguistic analogies—most famously "king is to queen as man is to woman"—led to a widespread assumption in natural language processing: that semantic relations can be modeled as constant vector differences. This "word2vec conjecture" suggests that for any semantic relation $R$, there exists a vector $v_R$ such that if $x$ and $y$ are in relation $R$, then $v(y) - v(x) = v_R$.

While significant effort has been spent on improving how these vectors are learned from data, this paper investigates a more fundamental question: Is it mathematically possible for a standard vector space to represent every type of linguistic relationship? By applying tools from abstract algebra and group theory, we demonstrate that the linear nature of traditional vector spaces is fundamentally incompatible with the cyclic structures inherent in human language.

The Word2vec Conjecture and Linearity

The core of the word2vec conjecture is the idea of a group homomorphism. If we treat semantic relations as operations that can be composed, the vector representation should preserve this composition. For example, if $R$ is the relation "the next day," then applying $R$ twice should correspond to adding the vector $v_R$ twice.

However, this linearity encounters a problem with cyclic relations. Consider the days of the week. If we represent "the next day" as a vector $\mathbf{s}$, then starting at Monday and applying the relation seven times brings us back to Monday. In the language of vectors, this implies:

$$v(\text{Monday}) + 7\mathbf{s} = v(\text{Monday})$$

This simplifies to $7\mathbf{s} = \mathbf{0}$. In vector spaces over fields like the real ($\mathbb{R}$) or complex ($\mathbb{C}$) numbers—which have characteristic 0—the only way for a non-zero integer multiple of a vector to equal zero is if the vector itself is the zero vector ($\mathbf{s} = \mathbf{0}$).

Theorem 4.1: The Limitative Result

The paper formalizes this observation in Theorem 4.1. It proves that any semantic relation with a finite "order" (a cycle) must be represented by the zero vector to satisfy the axioms of a standard vector space.

If the relation vector $\mathbf{s}$ is zero, the model becomes degenerate: it cannot distinguish between "the day after" and "the same day." This result effectively falsifies the word2vec conjecture for any system involving cycles, as the vector space is forced to collapse the entire cycle into a single point to maintain mathematical consistency.

Case Studies in Semantic Cycles

The paper highlights several common linguistic structures that fall victim to this limitative result:

  1. Temporal Cycles: Days of the week (order 7), months of the year (order 12), and hours of the day (order 24) all require cyclic representation that linear vectors cannot accommodate.
  2. Antonyms: The relation "antonym" is a cycle of order 2 (the antonym of an antonym is the original word). Under the word2vec conjecture, the vector for "antonym" would have to be zero, making "hot" and "cold" identical in that relational dimension.
  3. Monotonicity: The author notes that the log-probability math underlying word2vec is inherently monotonic. As one moves along a relational path, the values must strictly increase or decrease, preventing the "looping" required for cyclic concepts.

Discussion and Future Work

The failure of the word2vec conjecture reveals a fundamental "blind spot" in linear embeddings. Vector spaces are inherently "flat" and infinite, while language is dense with recursion, parity, and cycles.

To address these limitations, the paper suggests several paths forward:

  • Matrix Representations: Moving from vector addition to matrix multiplication. Invertible square matrices can represent complex group structures that vectors cannot.
  • Alternative Fields: Exploring complex numbers, where rotations (like $e^{2\pi i/k}$) can naturally model cycles of order $k$.
  • Non-commutative Geometry: Recognizing that many semantic relations do not commute (the order of operations matters), which suggests that the commutative nature of vector addition ($A+B = B+A$) is itself a limitation.

By establishing this limitative result, the paper encourages the field to look beyond simple linear shifts and toward more sophisticated geometric representations of meaning.

Summary updated

Abstract

Being inspired by the success of \texttt{word2vec} \citep{mikolov2013distributed} in capturing analogies, we study the conjecture that analogical relations can be represented by vector spaces. Unlike many previous works that focus on the distributional semantic aspect of \texttt{word2vec}, we study the purely \emph{representational} question: can \emph{all} semantic word-word relations be represented by differences (or directions) of vectors? We call this the word2vec conjecture and point out some of its desirable implications. However, we will exhibit a class of relations that cannot be represented in this way, thus falsifying the conjecture and establishing a limitative result for the representability of semantic relations by vector spaces over fields of characteristic 0, e.g., real or complex numbers.
BibTeX
@misc{dai2020word,
  author = {Falcon Z. Dai},
  title = {{Word2vec Conjecture and A Limitative Result}},
  year = {2020},
  eprint = {2010.12719},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.12719},
}
Figure