Context-Dependent Sense Embedding

Lin Qiu, Kewei Tu, Yong YuEMNLP 2016 · OpenReview

Abstract

Standard word embedding techniques represent each word with a single vector, which fails to capture the nuances of polysemous words. This paper introduces a novel probabilistic model for sense embedding that assigns distinct vectors to different meanings of the same word. By explicitly modeling the inter-dependencies between the senses of neighboring words in a sentence, the authors achieve a 13% relative gain over state-of-the-art models in word sense induction tasks.

Introduction

Word embedding models like Word2vec or GloVe are foundational in NLP but suffer from the "polysemy problem." When a word like "bank" is represented by a single vector, that vector becomes a blurred average of disparate meanings—such as a financial institution and a river edge.

Previous attempts at sense embeddings often relied on heuristic clustering or used global word embeddings as a starting point. Crucially, most prior work ignored the dependency between sense choices. In a sentence like "He cashed a check at the bank," the sense of "cashed" helps disambiguate "check," which in turn disambiguates "bank." This research argues for modeling these sense-to-sense relationships directly to improve embedding quality.

Methodology

The proposed probabilistic model is structured similarly to a high-order Hidden Markov Model (HMM). It defines an energy function for a sentence composed of two primary components:

  1. Word-Sense Match ($E_1$): Ensures an assigned sense is valid for the observed word.
  2. Sense Compatibility ($E_2$): Measures how well neighboring senses fit together. This "secret sauce" adapts standard architectures like CBOW and Skip-gram to calculate compatibility between sense vectors rather than word vectors.
The probabilistic model structure showing dependencies between senses ($s_i$) and their corresponding words ($w_i$).

Inference and Learning

To find the most likely senses for words in a new sentence, the model employs a dynamic programming algorithm (similar to the Viterbi algorithm) to identify the optimal sequence of senses that minimizes total energy. The model is trained on unannotated text (1.3 billion tokens from English Wikipedia) using an unsupervised, hard-EM style algorithm. In the E-step, the model estimates word senses; in the M-step, it updates embedding vectors to maximize the margin between "correct" senses and negative samples.

Evaluation and Results

The model was evaluated on two primary tasks: word similarity and word sense induction.

Word Similarity

On the Stanford’s Contextual Word Similarities (SCWS) dataset, the model outperformed other probabilistic sense models and remained competitive with complex clustering-based methods. By treating senses as "lexemes" rather than fixed synsets from an ontology like WordNet, the approach maintains flexibility.

Word Sense Induction (WSI)

The standout performance occurred in the SemEval-2013 Task 13. The model achieved a 13% relative gain in the harmonic mean (HM) of FBC and FNMI metrics, significantly outperforming the previous state-of-the-art, ICE-kmeans.

Performance comparison on the SemEval-2013 Word Sense Induction task, showing the significant gain in Harmonic Mean (HM).

Conclusion

This research demonstrates that modeling sense-to-sense dependencies is crucial for high-quality sense embeddings. By avoiding "polluted" global word embeddings and making global decisions for sentence sense sequences via dynamic programming, the model achieves superior results in disambiguation tasks. Future work will focus on automating sense inventory discovery and exploring "soft" EM assignments to further refine the learning process.

Summary updated

BibTeX
@inproceedings{qiu2016context,
  author = {Lin Qiu and Kewei Tu and Yong Yu},
  title = {{Context-Dependent Sense Embedding}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  url = {https://openreview.net/forum?id=SkNnvGfdZB},
}
Figure