Multi-phase Word Sense Embedding Learning Using a Corpus and a Lexical Ontology

Qi Li, Tianshi Li, Baobao Chang

Abstract

Standard word embedding models typically assign a single vector to each word, failing to distinguish between multiple meanings of polysemous words. This paper proposes a novel supervised approach to learning word sense embeddings by leveraging dictionary definitions from lexical ontologies like WordNet. By training Recurrent Neural Networks (RNNs) to map compositional definitions to specific concept vectors, the method achieves high performance on rare word senses and enables "reverse dictionary" functionality. Evaluations on standard benchmarks demonstrate that this approach is both efficient and competitive with state-of-the-art corpus-based methods.

Introduction

In natural language processing, the ambiguity of polysemous and homonymous words remains a significant hurdle. Traditional models such as Word2Vec or GloVe represent each word with a single point in a high-dimensional space, effectively averaging all its meanings. For instance, the vector for "apple" might be a hybrid of a fruit and a technology company, biased toward the more frequent usage in the training corpus.

To address this, researchers have sought to learn individual embeddings for each word sense. However, standard techniques often require performing Word Sense Disambiguation (WSD) on massive text corpora, which is computationally intensive and prone to error propagation. Furthermore, rare senses often lack sufficient context in raw text to form robust representations. This paper introduces a bootstrapping framework that utilizes the structured semantic information in dictionary definitions to learn sense-specific embeddings directly.

Methodology

The proposed method treats word sense definitions as supervised training data. Since definitions explain semantics compositionally, a sequence-to-vector model can be trained to "understand" the definition and output the corresponding sense vector.

The training process follows a three-phase bootstrapping strategy:

  1. Monosemous Initialization: The model is first trained on words with a single meaning. An RNN reads the pre-trained embeddings of the definition words and is optimized to predict the target word's pre-trained vector.
  2. Multi-sense Expansion: For polysemous words, sense embeddings are initialized using synonyms or cosine similarity heuristics. The RNN then processes the definitions of these specific senses, jointly updating the network weights and the sense embeddings.
  3. Recursive Refinement: To further improve accuracy, the words within the definitions themselves are disambiguated. Instead of generic word embeddings, the model begins to use its own learned sense embeddings as inputs, creating a refinement loop.
RNN architecture for mapping a dictionary definition to a specific word sense vector.

Experimental Results

The model was evaluated on several standard benchmarks, including WordSim-353 for context-free similarity and Stanford’s Contextual Word Similarities (SCWS).

On the WordSim-353 dataset, the LSTM-based model achieved a Spearman correlation ($\rho \times 100$) of 75.0, representing a significant improvement over the standard Skip-gram baseline of 66.5. This performance is competitive with much more complex models trained on significantly larger corpora.

Comparison of Spearman correlation results on the WordSim-353 dataset across various systems.

Key Insights

  • Rare Sense Robustness: Unlike corpus-based methods that rely on frequency, this approach treats all senses equally by providing each with a definition. This levels the playing field for obscure meanings that rarely appear in raw text.
  • Reverse Dictionary Capability: The trained RNN can map arbitrary natural language descriptions to concept vectors. For example, the model can successfully map the description "a machine we use every day" to the sense vector for "computer" or "dishwasher."
  • Importance of Memory: LSTMs and GRUs significantly outperformed vanilla RNNs. This highlights that capturing long-term dependencies within a definition is crucial for semantic compositionality.

Conclusion

By shifting the focus from raw corpora to curated lexical ontologies, this research provides a highly efficient and effective path toward sense-aware word representations. While dependent on the availability of high-quality dictionaries, the method offers a powerful tool for disambiguation and semantic search tasks, particularly for low-frequency word senses.

Summary updated

BibTeX
@inproceedings{li2016multi,
  author = {Qi Li and Tianshi Li and Baobao Chang},
  title = {{Multi-phase Word Sense Embedding Learning Using a Corpus and a Lexical Ontology}},
  booktitle = {NLPCC/ICCPOL},
  year = {2016},
  doi = {10.1007/978-3-319-50496-4\_19},
}
Figure