Abstract
We present an unsupervised method to generate Word2Sense word embeddings that are interpretable — each dimension of the embedding space corresponds to a fine-grained sense, and the non-negative value of the embedding along the j-th dimension represents the relevance of the j-th sense to the word. The underlying LDA-based generative model can be extended to refine the representation of a polysemous word in a short context, allowing us to use the embeddings in contextual tasks. On computational NLP tasks, Word2Sense embeddings compare well with other word embeddings generated by unsupervised methods. Across tasks such as word similarity, entailment, sense induction, and contextual interpretation, Word2Sense is competitive with the state-of-the-art method for that task. Word2Sense embeddings are at least as sparse and fast to compute as prior art.
Introduction
In Natural Language Processing (NLP), word embeddings such as Word2vec and GloVe are foundational. However, these dense vectors are essentially "black boxes"—individual dimensions lack inherent meaning. Furthermore, they typically assign a single vector to each word, failing to capture polysemy effectively. For instance, a single vector for "bank" must conflate its financial and geographical meanings. While multi-prototype models exist, they often struggle with computational efficiency or arbitrarily limit the number of senses per word.
Word2Sense addresses these issues by representing words as sparse vectors where each dimension corresponds to a specific, fine-grained sense. This approach provides human-interpretable coordinates and a natural mechanism for handling polysemy.
Methods
The core of Word2Sense is an unsupervised generative model for word co-occurrence based on Latent Dirichlet Allocation (LDA).
- Senses as Distributions: A "sense" is defined as a probability distribution over the entire vocabulary. For example, a "cricket" sense assigns high probability to words like wicket, bowler, and innings.
- Generative Process: To generate a context around a target word $w$, the model samples a sense $z$ from the word's sense distribution $\theta_w$, and then samples context words from that sense.
- Inference and Sparsity: By utilizing WarpLDA, the authors efficiently recover thousands of fine-grained senses from a 3.5-billion-token corpus (UKWAC and Wackypedia). The resulting embeddings are sparse, with a word's representation being its weights across these global senses.
- Contextual Disambiguation: The model can be extended to WordCtx2Sense using a Multiplicative Weight Update (MWU) algorithm to refine a word's embedding based on its immediate context, effectively selecting the appropriate sense for a given sentence.
Results and Discussion
Interpretability
A key advantage of Word2Sense is its inherent interpretability. Unlike dense embeddings, the dimensions of a Word2Sense vector correspond to distinct semantic clusters.
As shown in Table 1, the model successfully identifies various senses for words like "Cell" (ranging from biological to telecommunications) and "Apple" (from technology to fruit). Human evaluations confirmed this, with judges identifying "intruder" words in sense groups with 75% precision.
Performance on NLP Tasks
Interpretability does not come at the cost of performance. Word2Sense remains competitive with state-of-the-art dense models across various benchmarks.
In word similarity tasks (Table 2), Word2Sense achieved a Spearman correlation of 0.374 on the Rare Words (RW) dataset, outperforming Word2vec (0.365). It also showed strong results in word entailment, reaching an F1 score of 0.798, and in word sense induction, where it nearly doubled the performance of previous models on the Semeval-2010 task.
Conclusion
Word2Sense demonstrates that word embeddings can be both high-performing and human-readable. By leveraging a global pool of senses and an efficient LDA-based framework, it provides a scalable solution for interpretable NLP. While massive transformer models like BERT offer higher raw accuracy, Word2Sense provides a computationally efficient and transparent alternative, particularly valuable for tasks requiring semantic clarity and entailment reasoning.
Summary updated