Breaking Sticks and Ambiguities with Adaptive Skip-gram

Sergey Bartunov, Dmitry Kondrashkin, Anton Osokin, Dmitry P. Vetrov

Abstract

Recently proposed Skip-gram model is a powerful method for learning high-dimensional word representations that capture rich semantic relationships between words. However, Skip-gram as well as most prior work on learning word representations does not take into account word ambiguity and maintain only single representation per word. Although a number of Skip-gram modifications were proposed to overcome this limitation and learn multi-prototype word representations, they either require a known number of word meanings or learn them using greedy heuristic approaches. In this paper we propose the Adaptive Skip-gram model which is a nonparametric Bayesian extension of Skip-gram capable to automatically learn the required number of representations for all words at desired semantic resolution. We derive efficient online variational learning algorithm for the model and empirically demonstrate its efficiency on word-sense induction task.

Introduction

Standard word embedding models, such as the widely used Skip-gram, typically assign a single vector representation to each word. This "one-word, one-vector" approach inherently fails to capture word ambiguity (polysemy). For instance, the word "apple" can denote a fruit or a technology company; forcing these distinct meanings into a single vector results in a "muddy" average or a representation dominated by the most frequent sense. While previous multi-prototype models attempted to address this, they often required a pre-specified number of senses or relied on ad-hoc greedy heuristics to split words.

Methods: The AdaGram Model

The authors propose AdaGram (Adaptive Skip-gram), a nonparametric Bayesian extension of Skip-gram. The core innovation is the use of a Dirichlet Process (DP), specifically its "stick-breaking" construction, to handle an unknown number of word senses. This allows the model to theoretically support an infinite number of senses, allocating only as many as the data justifies.

A key hyperparameter, $\alpha$ (semantic resolution), controls the granularity of the discovered meanings. Higher values of $\alpha$ encourage the model to identify more specific sub-meanings. To ensure scalability to large corpora like Wikipedia, the authors developed an online stochastic variational inference (SVI) algorithm, enabling the model to learn in a single pass.

Distribution of prototypes and frequency scaling.

As shown in the figure above, the number of prototypes discovered for a word scales roughly logarithmically with its frequency. This adaptive complexity is a hallmark of the model: common, polysemous words receive multiple prototypes, while rare or unambiguous words remain simple.

Evaluation and Results

AdaGram was evaluated on several benchmarks, including SemEval Word Sense Induction (WSI) tasks and a newly introduced dataset, WWSI (Wikipedia Word-sense Induction). WWSI consists of 188 target words and over 36,000 contexts derived from Wikipedia disambiguation pages, providing a robust test for multi-prototype models.

Performance comparison on WWSI task.

The results demonstrate that AdaGram consistently outperforms existing multi-prototype baselines like MSSG and NP-MSSG. On the WWSI dataset, AdaGram achieved an Adjusted Rand Index (ARI) of 0.286, a significant improvement over the best baseline of 0.194. The figure above illustrates this superiority through Recall@K and Precision@r curves, where AdaGram ($\alpha=0.15$) maintains a clear lead.

Qualitatively, the model successfully disentangles meanings for words like "python" (programming vs. biology) and "bow" (weaponry vs. nautical vs. decorative). Furthermore, AdaGram achieved better test log-likelihood than standard Skip-gram, indicating that modeling polysemy improves the overall predictive power of the language model.

Conclusion

AdaGram provides a principled, automated framework for learning multi-prototype word representations. By leveraging nonparametric Bayesian methods, it avoids the pitfalls of manual sense specification and greedy heuristics. While it incurs a higher computational cost than standard Skip-gram, its ability to capture semantic nuances makes it a powerful tool for tasks requiring deep linguistic understanding. Future work may explore its application in downstream tasks like machine translation or information retrieval, where disambiguation is critical.

Summary updated

Abstract

Recently proposed Skip-gram model is a powerful method for learning high-dimensional word representations that capture rich semantic relationships between words. However, Skip-gram as well as most prior work on learning word representations does not take into account word ambiguity and maintain only single representation per word. Although a number of Skip-gram modifications were proposed to overcome this limitation and learn multi-prototype word representations, they either require a known number of word meanings or learn them using greedy heuristic approaches. In this paper we propose the Adaptive Skip-gram model which is a nonparametric Bayesian extension of Skip-gram capable to automatically learn the required number of representations for all words at desired semantic resolution. We derive efficient online variational learning algorithm for the model and empirically demonstrate its efficiency on word-sense induction task.
BibTeX
@inproceedings{bartunov2016breaking,
  author = {Sergey Bartunov and Dmitry Kondrashkin and Anton Osokin and Dmitry P. Vetrov},
  title = {{Breaking Sticks and Ambiguities with Adaptive Skip-gram}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2016},
  url = {https://openreview.net/forum?id=umuNzfMpl-},
}
Figure