Introduction
Standard word embedding models, such as Word2Vec, typically generate dense vectors where information is distributed across all dimensions. While effective for many downstream tasks, these "black box" representations lack human interpretability, as individual dimensions rarely correspond to discrete semantic concepts. Previous attempts to introduce sparsity often relied on computationally expensive post-processing techniques that do not scale well to large datasets.
This paper introduces a novel approach to learning sparse word embeddings directly during the training phase. By incorporating $\ell_1$ regularization into the Continuous Bag of Words (CBOW) architecture and utilizing the Regularized Dual Averaging (RDA) algorithm, the authors produce vectors that are over 90% sparse while matching or exceeding the performance of traditional dense models.
Methodology
The proposed method modifies the standard CBOW model through three primary components:
- $\ell_1$ Regularization: An $\ell_1$ penalty is added to the word vectors to encourage sparsity, effectively pushing many parameters toward zero.
- Regularized Dual Averaging (RDA): Unlike standard Stochastic Gradient Descent (SGD), which often fails to produce exact zeros in an online setting due to gradient noise, RDA tracks the running average of gradients. This allows the optimization process to "snap" weights to zero, achieving true sparsity.
- Negative Sampling: To maintain computational efficiency and scalability for billion-word corpora, the model employs negative sampling to approximate the softmax function.
Experimental Results
The model was trained on a 1-billion-word Wikipedia dump and evaluated across several word similarity and analogy tasks, including WordSim-353, SimLex-999, and the Rare Word (RW) dataset.
Performance and Sparsity
The Sparse CBOW model achieved a sparsity level of 90.06%. Despite this significant reduction in active parameters, it outperformed dense counterparts on several benchmarks. Notably, on the challenging SimLex-999 dataset, Sparse CBOW achieved a score of 44.47, compared to 38.82 for dense CBOW.
Stability and Scalability
A key finding was the stability of the sparse model relative to vector dimensionality. While the performance of dense CBOW tends to degrade as dimensionality increases beyond a certain point (e.g., 300), Sparse CBOW maintains high performance even at 1000 dimensions. This suggests that the $\ell_1$ regularization acts as an effective safeguard against overfitting in high-dimensional spaces.
Interpretability and Discussion
The sparse representations offer significantly better interpretability. In "word intrusion" tasks, dimensions in the Sparse CBOW model were found to be highly coherent. For instance, specific dimensions were observed to activate exclusively for clusters of related terms, such as statistical learning concepts (poisson, markov, bayesian).
While Sparse CBOW excels in semantic similarity, it slightly trails the best dense models in purely syntactic analogies, such as verb tense variations. This suggests that while sparsity captures semantic concepts effectively, dense representations may still hold an advantage for certain linguistic regularities. Future work may explore the addition of non-negativity constraints to further enhance interpretability.
Summary updated