Rehabilitation of Count-based Models for Word Vector Representations

Rémi Lebret, Ronan Collobert

Abstract

Recent works on word representations mostly rely on predictive models. Distributed word representations (aka word embeddings) are trained to optimally predict the contexts in which the corresponding words tend to appear. Such models have succeeded in capturing word similarities as well as semantic and syntactic regularities. Instead, we aim at reviving interest in a model based on counts. We present a systematic study of the use of the Hellinger distance to extract semantic representations from the word co-occurrence statistics of large text corpora. We show that this distance gives good performance on word similarity and analogy tasks, with a proper type and size of context, and a dimensionality reduction based on a stochastic low-rank approximation. Besides being both simple and intuitive, this method also provides an encoding function which can be used to infer unseen words or phrases. This becomes a clear advantage compared to predictive models which must train these new words.

Introduction

In the early 2010s, the Natural Language Processing (NLP) world shifted heavily toward "predictive" models—neural networks like Skip-gram and CBOW that learn word meanings by trying to predict a word's neighbors. These models became the gold standard, often leaving older "count-based" models (which simply count how often words appear together) behind. However, predictive models are computationally expensive, require complex tuning, and are "closed" systems that struggle with new vocabulary. This paper "rehabilitates" count-based models, proving that with the right mathematical refinements, they remain top-tier contenders.

Methods

The authors propose a refined pipeline for building word representations from co-occurrence statistics:

  1. Probability Mapping: Calculating the conditional probability of a context word given a target word.
  2. The Hellinger Transformation: Applying a square-root transformation to probabilities to use the Hellinger distance, which is better suited for comparing discrete distributions than Euclidean distance.
  3. Stochastic Low-Rank Approximation (SLRA): A novel dimensionality reduction technique using backpropagation and stochastic gradient descent. This avoids the memory constraints and numerical fragility of standard PCA/SVD on massive, sparse matrices.
  4. Flexible Context Windows: Experimenting with symmetric and asymmetric windows of varying sizes to capture different linguistic properties.

Key Insights

A major discovery was the Window Size Trade-off: small windows (size 1) are excellent for capturing syntax, while large windows (size 10) are necessary for semantics. Additionally, the authors found that Rare Words Matter: discarding the most frequent stop-words from the context dictionary improves performance, as rare words provide more specific signals about a word's meaning.

Figure 6: Accuracy on Mikolov's syntactic (left) and semantic (right) datasets. Small windows (sym1) are optimal for syntax, while larger windows (sym10) are required for semantic tasks.

Unlike predictive models, this approach provides an explicit encoding function. To represent a new phrase, one simply counts its neighbors and applies the existing projection matrix, avoiding the need for retraining.

Results

The SLRA model proved highly competitive with state-of-the-art predictive models like GloVe and Word2Vec. On similarity tasks (WS-353 and RG-65), it achieved Spearman correlation scores of 0.57–0.60. On analogy tasks, the model reached 61% accuracy on syntactic tasks and 62.5% on semantic tasks.

Figure 4: Relatedness score on the Stanford Rare Word (RW) dataset. Performance is enhanced by selecting specific context word frequency ranges, supporting the claim that rare words provide better signals.

The stochastic approach to dimensionality reduction (SLRA) significantly outperformed standard Hellinger PCA (HPCA), proving more robust for high-dimensional linguistic data.

Conclusion

This work demonstrates that count-based models, when combined with the Hellinger distance and stochastic low-rank approximation, are a viable and efficient alternative to neural predictive models. They offer comparable accuracy on standard benchmarks while providing superior flexibility for handling new words and phrases. Future work may extend this framework to even larger corpora and more complex phrase-level embeddings.

Summary updated

Abstract

Recent works on word representations mostly rely on predictive models. Distributed word representations (aka word embeddings) are trained to optimally predict the contexts in which the corresponding words tend to appear. Such models have succeeded in capturing word similarties as well as semantic and syntactic regularities. Instead, we aim at reviving interest in a model based on counts. We present a systematic study of the use of the Hellinger distance to extract semantic representations from the word co-occurence statistics of large text corpora. We show that this distance gives good performance on word similarity and analogy tasks, with a proper type and size of context, and a dimensionality reduction based on a stochastic low-rank approximation. Besides being both simple and intuitive, this method also provides an encoding function which can be used to infer unseen words or phrases. This becomes a clear advantage compared to predictive models which must train these new words.
BibTeX
@inproceedings{lebret2015rehabilitation,
  author = {Rémi Lebret and Ronan Collobert},
  title = {{Rehabilitation of Count-based Models for Word Vector Representations}},
  booktitle = {CICLing},
  year = {2015},
  url = {https://openreview.net/forum?id=q1ElbTTTfzL},
}
Figure