Abstract
The recently proposed Skip-gram model is a powerful method for learning high-dimensional word representations that capture rich semantic relationships between words. However, Skip-gram as well as most other word embedding models assume a single vector for each word, which is problematic for polysemous words. We present AdaGram, a non-parametric Bayesian extension of Skip-gram that automatically learns the required number of representations for each word. Our model is based on the Dirichlet Process and can be trained using efficient online variational inference. We demonstrate that AdaGram outperforms existing multi-prototype models on word sense induction tasks and provides a more nuanced understanding of word meanings.
Introduction: The Polysemy Problem
Standard word embedding models like Word2Vec (Skip-gram and CBOW) assign exactly one vector to each word. This is a fundamental limitation because natural language is rife with polysemy (related meanings, e.g., "bank" of a river vs. "bank" for money) and homonymy (unrelated meanings). Forcing these distinct semantic roles into a single vector results in a "mushy" average that fails to represent any specific meaning accurately, or a vector dominated by the most frequent usage. While previous attempts to address this used pre-defined sense counts or greedy heuristics, they often lacked flexibility or theoretical rigor.
AdaGram: Adaptive Skip-gram
AdaGram addresses these issues by incorporating a Bayesian non-parametric framework into the Skip-gram architecture.
- Dirichlet Process (Stick-Breaking): Instead of a fixed number of senses, AdaGram uses a Dirichlet Process. This "stick-breaking" process allows the model to dynamically allocate new prototypes as it encounters more data, theoretically allowing for an infinite number of meanings.
- Latent Sense Variables: The model introduces a latent variable $z$ to represent the specific sense used in a given context.
- Semantic Resolution ($\alpha$): The hyperparameter $\alpha$ controls the granularity of sense discovery. A higher $\alpha$ encourages the model to identify more specific, fine-grained meanings.
- Online Variational Inference: To maintain the scalability of the original Skip-gram, the authors developed an efficient online learning algorithm, enabling the model to process massive datasets and update word senses on the fly.
Experimental Setup
The model was trained on a 2010 snapshot of English Wikipedia (approx. 990 million tokens). Evaluation focused on Word Sense Induction (WSI) using standard benchmarks (SemEval-2007, 2010, 2013) and a new, larger dataset called WWSI (Wikipedia Word-sense Induction), which includes 188 target words and over 36,000 contexts.
Insights into Sense Discovery
A key finding is that AdaGram naturally respects Zipf’s Law. The number of prototypes discovered for a word scales logarithmically with its frequency. Rare words typically retain a single prototype, while common, ambiguous words like "run" or "light" sprout multiple representations as more evidence of varied usage appears.
As shown in the figure above, the granularity parameter $\alpha$ acts as a "zoom lens." At low $\alpha$, the model identifies broad categories; at high $\alpha$, it captures subtle sub-meanings, such as distinguishing between "light" in a physical sense versus a military context.
Performance and Results
Qualitatively, AdaGram successfully separates distinct meanings for words like "python" (programming language vs. snake) and "apple" (fruit vs. technology company). Quantitatively, AdaGram significantly outperforms concurrent state-of-the-art models like MSSG and NP-MSSG on WSI tasks. On the WWSI dataset, AdaGram achieved an Adjusted Rand Index (ARI) of 0.286, compared to 0.194 for its closest competitor.
Furthermore, AdaGram achieved better predictive log-likelihood on held-out Wikipedia text than the original Skip-gram, demonstrating that modeling multiple senses improves the model's overall "understanding" of natural language.
Discussion and Future Work
While efficient, AdaGram is roughly $T$ times slower than the original Skip-gram (where $T$ is the maximum allowed senses). Additionally, high $\alpha$ values can lead to sense redundancy, where semantically similar prototypes are created. The next major challenge is the seamless integration of these multi-prototype embeddings into downstream NLP tasks like machine translation, which requires accurately selecting the correct sense vector for a given context.
Summary updated