Abstract
Standard word embedding techniques represent each word with a single vector, which fails to capture the nuances of polysemous words. This paper introduces a novel probabilistic model for sense embedding that assigns distinct vectors to different meanings of the same word. By explicitly modeling the inter-dependencies between the senses of neighboring words in a sentence, the authors achieve a 13% relative gain over state-of-the-art models in word sense induction tasks.
Introduction
Word embedding models like Word2vec or GloVe are foundational in NLP but suffer from the "polysemy problem." When a word like "bank" is represented by a single vector, that vector becomes a blurred average of disparate meanings—such as a financial institution and a river edge.
Previous attempts at sense embeddings often relied on heuristic clustering or used global word embeddings as a starting point. Crucially, most prior work ignored the dependency between sense choices. In a sentence like "He cashed a check at the bank," the sense of "cashed" helps disambiguate "check," which in turn disambiguates "bank." This research argues for modeling these sense-to-sense relationships directly to improve embedding quality.
Methodology
The proposed probabilistic model is structured similarly to a high-order Hidden Markov Model (HMM). It defines an energy function for a sentence composed of two primary components:
- Word-Sense Match ($E_1$): Ensures an assigned sense is valid for the observed word.
- Sense Compatibility ($E_2$): Measures how well neighboring senses fit together. This "secret sauce" adapts standard architectures like CBOW and Skip-gram to calculate compatibility between sense vectors rather than word vectors.
Inference and Learning
To find the most likely senses for words in a new sentence, the model employs a dynamic programming algorithm (similar to the Viterbi algorithm) to identify the optimal sequence of senses that minimizes total energy. The model is trained on unannotated text (1.3 billion tokens from English Wikipedia) using an unsupervised, hard-EM style algorithm. In the E-step, the model estimates word senses; in the M-step, it updates embedding vectors to maximize the margin between "correct" senses and negative samples.
Evaluation and Results
The model was evaluated on two primary tasks: word similarity and word sense induction.
Word Similarity
On the Stanford’s Contextual Word Similarities (SCWS) dataset, the model outperformed other probabilistic sense models and remained competitive with complex clustering-based methods. By treating senses as "lexemes" rather than fixed synsets from an ontology like WordNet, the approach maintains flexibility.
Word Sense Induction (WSI)
The standout performance occurred in the SemEval-2013 Task 13. The model achieved a 13% relative gain in the harmonic mean (HM) of FBC and FNMI metrics, significantly outperforming the previous state-of-the-art, ICE-kmeans.
Conclusion
This research demonstrates that modeling sense-to-sense dependencies is crucial for high-quality sense embeddings. By avoiding "polluted" global word embeddings and making global decisions for sentence sense sequences via dynamic programming, the model achieves superior results in disambiguation tasks. Future work will focus on automating sense inventory discovery and exploring "soft" EM assignments to further refine the learning process.
Summary updated