Learning Word Embeddings without Context Vectors

Alexey Zobnin, Evgenia ElistratovaACL 2019

TL;DR

This paper introduces "context-free SGNS" (cf-SGNS), a novel approach to learning word embeddings that eliminates the need for separate context vectors. By using an indefinite inner product, cf-SGNS achieves comparable performance to traditional methods while simplifying the model and potentially offering new linguistic interpretations.

What Problem Does This Paper Address?

Traditional word embedding methods like word2vec and fastText (specifically, the Skip-gram Negative Sampling or SGNS algorithm) learn two sets of vectors: one for words and one for their surrounding contexts. While effective, this duality raises questions about the necessity of separate context vectors and their relationship to word vectors. The paper observes that in practice, only word vectors are often used for downstream tasks like word similarity. This suggests a potential redundancy or an opportunity for a more unified representation. The authors aim to explore whether word and context vectors can be treated as equivalent, or at least closely related, and if this can lead to a more efficient and interpretable embedding model.

Methods

The core innovation lies in modifying the SGNS loss function. Instead of the standard dot product between word and context vectors, the authors propose using an indefinite inner product. This is achieved by introducing a diagonal matrix $D$ with entries of $\pm 1$. The loss function is then reformulated to use this indefinite inner product, effectively equating word and context vectors. This is justified by observations from Singular Value Decomposition (SVD) of symmetric matrices, where word and context matrices can be related by a diagonal matrix of signs. The proposed algorithm, termed "context-free SGNS" (cf-SGNS), learns only one set of vectors, which are then used in this indefinite inner product. The number of negative entries in $D$ (denoted by $q$) becomes a hyperparameter.

Datasets

Training Data: * English Wikipedia dump (subsampled to 100K articles, approximately 175M words, vocabulary size $\approx$ 312,000).

Evaluation Datasets: The paper evaluates word similarity using several standard English datasets:

  • MEN-3k
  • MTurk-287
  • RW-STANFORD
  • SIMLEX-999
  • SimVerb-3500
  • VERB-143
  • WS-353 (split into similarity and relatedness parts: WS-353-REL and WS-353-SIM)

Key Insights and Ideas

  • Equivalence of Word and Context Vectors: The paper provides theoretical grounding for treating word and context vectors as essentially the same, or at least derivable from each other through sign changes, by leveraging SVD properties of symmetric matrices.
  • Indefinite Inner Product as a Solution: The indefinite inner product is a mathematically sound way to model the relationship where vectors can be "similar" or "dissimilar" based on the sign of their components, mirroring the learned relationships in SGNS.
  • Simplification of Embedding Models: Eliminating separate context vectors reduces model complexity and memory footprint, while potentially retaining or even improving performance.
  • Linguistic Interpretation of Negative Components: The use of negative eigenvalues in SVD and negative entries in the $D$ matrix suggests that "negative" components in embeddings might have a specific linguistic interpretation, which warrants further investigation.

Key Results

  • Comparable Performance: The cf-SGNS algorithm achieves word similarity scores that are on par with or slightly lower than standard SGNS. For instance, in Table 3 (d=100), the average Spearman correlation for cf-SGNS with optimal $q$ (10 or 15) is around 0.438, very close to SGNS's 0.438.
  • Impact of Hyperparameter $q$: The performance of cf-SGNS is sensitive to the hyperparameter $q$ (the number of negative entries in $D$). $q=0$ (positive semidefinite approximation) yields the worst performance, while $q=10$ or $q=15$ generally gives the best results.
  • Projection Experiment: When cf vectors of dimension $100+q$ were projected to 100 "positive" components, the performance (Table 4) was slightly better than SGNS for $q \ge 5$.

Limitations and Future Work

  • Linguistic Interpretation: The paper acknowledges that a deeper understanding of the linguistic interpretation of the "negative" components in the learned word vectors is needed.
  • Implicit Regularization: The phenomenon of implicit regularization in SGNS and its connection to the proposed cf-SGNS model requires further investigation.
  • Hyperparameter Tuning: The optimal value of $q$ needs to be determined, and its impact across different datasets and tasks could be explored more extensively.
  • Comparison with other methods: While compared to SGNS, a broader comparison with other state-of-the-art embedding techniques could be beneficial.

Summary updated

BibTeX
@inproceedings{zobnin2019learning,
  author = {Alexey Zobnin and Evgenia Elistratova},
  title = {{Learning Word Embeddings without Context Vectors}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}

Loading related content...

Figure