Abstract
Word embedding is a powerful tool in natural language processing. In this paper we consider the problem of word embedding composition --- given vector representations of two words, compute a vector for the entire phrase. We give a generative model that can capture specific syntactic relations between words. Under our model, we prove that the correlations between three words (measured by their PMI) form a tensor that has an approximate low rank Tucker decomposition. The result of the Tucker decomposition gives the word embeddings as well as a core tensor, which can be used to produce better compositions of the word embeddings. We also complement our theoretical results with experiments that verify our assumptions, and demonstrate the effectiveness of the new composition method.
Introduction
While standard word embeddings (e.g., Word2Vec, GloVe) effectively capture individual word semantics, representing the meaning of multi-word phrases remains a challenge. The common practice of vector addition ($v_a + v_b$) often fails to capture nuances in phrases where the combined meaning is more than the sum of its parts, such as "complex number" or "civil war." While some models assign a unique matrix to every adjective to handle composition, such approaches are computationally expensive and prone to overfitting. This paper introduces a mathematically principled generative model that incorporates syntax into composition using tensor decomposition.
Generative Models for Syntax
The proposed framework builds upon the RAND-WALK model, which treats text generation as a discourse vector moving through a latent space. The authors extend this into the Syntactic RAND-WALK model, which specifically accounts for pairs of syntactically related words (e.g., adjective-noun or verb-object).
In the Syntactic RAND-WALK model, the generation of word pairs is mediated by a universal composition tensor $T$. This allows the model to capture complex interactions between words without requiring a unique matrix for every vocabulary item.
Tensor Decomposition and Composition
The authors establish a theoretical link between high-order word statistics and the generative model. They prove that the Pointwise Mutual Information between three words (PMI3)---specifically two words in a phrase plus a neighboring word---can be approximated by a low-rank Tucker decomposition. This decomposition yields both standard word embeddings and a core tensor $T$.
Based on the model, the "correct" way to represent a phrase $(a, b)$ is derived as: $$\hat{c} \approx v_a + v_b + T(v_a, v_b, \cdot)$$ This formula represents phrase composition as the traditional additive model plus a "correction term" provided by the tensor $T$, which captures the specific syntactic relationship.
Experimental Results
The model was trained on an English Wikipedia dump and evaluated using the Stanford Dependency Parser to identify syntactic pairs.
Phrase Similarity and Qualitative Analysis
On the Mitchell and Lapata (2010) phrase similarity dataset, the tensor-based method consistently outperformed simple addition across various base embeddings (RAND-WALK, GloVe, and CBOW). Qualitatively, the method showed superior precision in neighbor retrieval. For example, when querying "complex numbers," the additive method returned general mathematical terms, whereas the tensor method successfully retrieved specific related concepts like "eigenvalues" and "quaternions."
Sentiment Analysis
While the method excelled at capturing semantic meaning in similarity tasks, its impact on sentiment analysis (using movie review datasets) was less significant. This suggests that the syntactic nuances captured by the tensor are more critical for semantic precision than for broad sentiment polarity.
Conclusion
This work provides a rare bridge between generative probabilistic models and tensor algebra. By demonstrating that phrase meaning has a specific geometric structure (Tucker rank), the authors offer a parameter-efficient way to incorporate syntax into word embeddings. Future work may extend this approach from word pairs to more complex structures like subject-verb-object triples.
Summary updated