The Influence of Down-Sampling Strategies on SVD Word Embedding Stability

Johannes Hellrich, Bernd Kampe, Udo HahnNAACL 2019

TL;DR

This paper investigates how different down-sampling strategies affect the stability and accuracy of word embeddings. They introduce a modified SVDPPMI algorithm, SVDWPPMI, which uses weighted down-sampling, achieving superior stability and competitive accuracy compared to existing methods. This work is crucial for reliable downstream applications of word embeddings, especially in fields like historical linguistics and medical informatics.

What Problem Does This Paper Address?

Word embeddings, dense vector representations of words, are fundamental to many Natural Language Processing (NLP) tasks. However, a growing concern in the field is the stability or reliability of these embeddings – how consistent are the representations when trained multiple times on the same data? This inconsistency can lead to unreliable interpretations and hinder reproducibility. Previous work has explored how down-sampling strategies (techniques to reduce the impact of frequent words or emphasize nearby words) affect embedding accuracy (performance on tasks like word similarity and analogy). This paper, however, focuses on the under-explored impact of these same down-sampling strategies on embedding stability, aiming to explain conflicting reports in the literature and propose a solution for more robust embeddings.

Methods

The authors compare five algorithm variants: GLOVE, SGNS, SVDPPMI (with and without probabilistic down-sampling), and their proposed SVDWPPMI.

  • SVDPPMI (Singular Value Decomposition of Positive Pointwise Mutual Information): This algorithm first constructs a word-context matrix, transforms it into a PPMI matrix, and then applies SVD to reduce dimensionality.
  • Down-sampling Strategies: The paper details two types of down-sampling:
  • Distance-based (df): Emphasizes nearby words within a context window.
  • Frequency-based (ff): Reduces the impact of high-frequency words. These factors can be applied probabilistically (each co-occurrence is processed with a certain probability) or by weighting (a fraction of the count is added).

  • SVDWPPMI (Weighted SVDPPMI): This is the paper's novel contribution. It modifies SVDPPMI by replacing probabilistic down-sampling with a weighted approach, using fractional counts based on the combined distance and frequency factors.

  • Stability Measurement: Stability is quantified using the Jaccard coefficient (j@n) between sets of the n most similar words to a set of anchor words, calculated across multiple independently trained models.
  • Accuracy Measurement: Standard Spearman rank correlation is used to compare embedding similarity judgments with human annotations on four psycholinguistic test sets (MEN, MTurk, SimLex-999, WS-353) and analogy tasks (Google, MSR).

Datasets

The study uses three English corpora for training and evaluation:

  • COHA (Corpus of Historical American English): Approximately 28 million tokens.
  • NEWS (English News Crawl Corpus): Approximately 550 million tokens.
  • WIKI (Wikipedia Corpus): Approximately 1.7 billion tokens.

These corpora are larger than those typically used in stability studies, allowing for a more comprehensive evaluation. The authors also used random subsamples of these corpora to simulate variations in training data and assess their impact on stability.

Key Insights and Ideas

  • Down-sampling is a critical factor for stability: The choice of down-sampling strategy, often considered a minor detail, has a profound impact on the stability of SVDPPMI-type embeddings.
  • Probabilistic down-sampling can hurt stability: Conflicting reports on SVDPPMI stability can be explained by differences in whether probabilistic down-sampling was used.
  • Weighting offers a stable alternative: The proposed SVDWPPMI, which uses weighted down-sampling, achieves significantly higher stability.
  • Corpus size matters for stability: Generally, larger corpora lead to more stable embeddings, although GLOVE shows an inverse relationship.
  • Subsampling degrades stability: Randomly subsampling corpora negatively impacts embedding stability, highlighting the importance of consistent training data.
  • GLOVE's trade-off: GLOVE demonstrates high stability on small corpora but suffers from lower accuracy, and its stability is not as robust on larger datasets compared to other methods.

Key Results

  • SVDWPPMI achieves near-perfect stability: When trained on non-subsampled corpora, SVDWPPMI and SVDPPMI without down-sampling achieve perfect stability (j@10 = 1.0).
  • SVDWPPMI outperforms others: SVDWPPMI is found to be significantly better than SVDPPMI (with or without probabilistic down-sampling) and GLOVE in terms of stability.
  • Competitive accuracy: SVDWPPMI achieves accuracy comparable to SGNS and is significantly better than SVDPPMI without down-sampling.
  • Corpus size impact: Larger corpora generally improve accuracy, but the best performance for some tasks (MEN, MTurk, MSR) was observed on the NEWS corpus, not the largest WIKI corpus.
  • Subsampling's negative effect: Subsampling corpora led to a decrease in both accuracy and stability for most algorithms.

Limitations and Future Work

  • Memory Consumption: A noted limitation of SVDPPMI variants, including SVDWPPMI, is their relatively high memory consumption during training.
  • Hyperparameter Tuning: The study used hyperparameters largely following Levy et al. (2015). Further research could explore the impact of different hyperparameter settings.
  • Downstream Task Evaluation: While accuracy was measured on similarity and analogy tasks, the authors suggest investigating the effect of down-sampling and stability on actual downstream NLP tasks.
  • Contextualized Embeddings: The paper acknowledges that modern contextualized embedding models (like BERT and ELMO) are also probabilistic and likely affected by stability issues. Adapting their evaluation strategy to these models by averaging token-specific representations is a promising future direction.

Summary updated

BibTeX
@inproceedings{hellrich2019influence,
  author = {Johannes Hellrich and Bernd Kampe and Udo Hahn},
  title = {{The Influence of Down-Sampling Strategies on SVD Word Embedding Stability}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure