Deconstructing word embedding algorithms

Kian Kenyon-Dean, Edward Newell, Jackie Chi Kit Cheung

Abstract

Word embeddings are reliable feature representations of words used to obtain high quality results for various NLP applications. Uncontextualized word embeddings are used in many NLP tasks today, especially in resource-limited settings where high memory capacity and GPUs are not available. Given the historical success of word embeddings in NLP, we propose a retrospective on some of the most well-known word embedding algorithms. In this work, we deconstruct Word2vec, GloVe, and others, into a common form, unveiling some of the common conditions that seem to be required for making performant word embeddings. We believe that the theoretical findings in this paper can provide a basis for more informed development of future models.

Introduction

In the mid-2010s, the NLP landscape was transformed by uncontextualized word embeddings—algorithms that map words to dense vector spaces. While models such as Word2vec (SGNS) and GloVe became industry standards, they were largely developed as distinct heuristic approaches. This research addresses a fundamental theoretical gap: why do these diverse algorithms perform so effectively, and what is the underlying mathematical framework they share? By deconstructing these models, we aim to provide a rigorous foundation that explains their success and informs the design of future language models.

Methods: The Low Rank Embedder Framework

The researchers propose the low rank embedder framework, which conceptualizes word embedding as a matrix reconstruction problem. The objective is to learn two maps: one for "focal" words (vectors) and one for "context" words (covectors).

The core of the analysis relies on the characteristic gradient. By examining the derivative of the loss function with respect to the model's internal parameters, the researchers can compare models on a unified basis. Five major algorithms are analyzed through this lens:

  1. SVD (Singular Value Decomposition): The classic linear algebra approach.
  2. SGNS (Skip-gram with Negative Sampling): The core of Word2vec.
  3. FastText: An extension of Word2vec utilizing subword information.
  4. GloVe: A model combining global statistics with factorized representations.
  5. LDS (Latent Discourse Space): A generative model based on random walks.

Empirical validation was performed using a Wikipedia 2018 corpus to demonstrate that the distribution of Pointwise Mutual Information (PMI) values is approximately normal and to verify theoretical predictions regarding GloVe’s learned biases.

Key Insights and Ideas

The analysis reveals several critical commonalities across successful embedding algorithms:

  • PMI as the Universal Target: Every analyzed algorithm—whether based on neural sampling (Word2vec) or matrix factorization (GloVe)—fundamentally attempts to approximate the Pointwise Mutual Information (PMI) between words. PMI measures the likelihood of two words appearing together compared to random chance.
  • The "Tempered" Multiplier: A vital design feature is the modulation of the loss gradient. The "multiplier" in the gradient is consistently a sub-linear (tempered) version of word frequency. This balancing act prevents frequent words (like "the" or "of") from dominating the learning process, ensuring the model remains sensitive to signals from rarer words.
  • PMI as a Normalization Trick: The effectiveness of PMI stems from its ability to transform highly imbalanced raw co-occurrence counts into an approximately normal distribution, which is significantly easier for machine learning models to optimize.
Distribution of PMI values and the relationship between GloVe biases and word frequency.
  • GloVe’s Biases as Statistics: The research provides a theoretical explanation for GloVe’s bias terms, showing that these learned parameters empirically capture the unigram counts required to transform raw co-occurrence data into PMI.

Key Results

The primary result is the unification of disparate algorithms into a single mathematical form. The authors demonstrate that for all these models, the gradient takes the form of (multiplier) · (difference), where the multiplier handles data imbalance and the difference drives the model toward the PMI target.

Table 1: Comparison of embedding models within the low rank embedder framework.

Specific findings include:

  • SGNS: Proven to approximate $PMI(i, j) - \ln k$, where $k$ is the number of negative samples.
  • GloVe: Proven to approximate $PMI(i, j)$ through its bilinear structure and bias terms.

Conclusion

While this study focuses on uncontextualized embeddings, its findings offer valuable conjectures for modern contextualized models like BERT and GPT. The authors suggest that deep contextualized models might benefit from explicitly incorporating PMI statistics into their training objectives and that sub-linear scaling for frequent word pairs remains a critical design choice. Furthermore, the use of dual representations (vectors and covectors) continues to be a powerful architectural strategy for capturing linguistic relationships.

Summary updated

Abstract

Word embeddings are reliable feature representations of words used to obtain high quality results for various NLP applications. Uncontextualized word embeddings are used in many NLP tasks today, especially in resource-limited settings where high memory capacity and GPUs are not available. Given the historical success of word embeddings in NLP, we propose a retrospective on some of the most well-known word embedding algorithms. In this work, we deconstruct Word2vec, GloVe, and others, into a common form, unveiling some of the common conditions that seem to be required for making performant word embeddings. We believe that the theoretical findings in this paper can provide a basis for more informed development of future models.
BibTeX
@inproceedings{kenyondean2020deconstructing,
  author = {Kian Kenyon-Dean and Edward Newell and Jackie Chi Kit Cheung},
  title = {{Deconstructing word embedding algorithms}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=N9g00iYIzA},
}

Loading related content...

Figure