Learning Word Sense Embeddings from Word Sense Definitions

Qi Li, Tianshi Li, Baobao Chang

TL;DR

This paper introduces a novel method for learning word sense embeddings directly from dictionary definitions using Recurrent Neural Networks (RNNs). This approach bypasses the need for large, potentially noisy corpora for training, leading to more efficient and potentially higher-quality embeddings, especially for rare word senses.

What Problem Does This Paper Address?

Traditional word embeddings, like those from Skip-gram or GloVe, assign a single vector representation to each word. This is problematic for polysemous words (words with multiple meanings) and homonymous words (words that sound alike but have different meanings), as a single vector cannot adequately capture their distinct senses. While researchers have proposed learning separate embeddings for each word sense, these methods typically rely on training on large text corpora. This corpus-based approach faces challenges: it can be computationally expensive, and the accuracy of word sense induction (WSI) or word sense disambiguation (WSD) on the corpus can be unreliable, introducing errors into the learned embeddings. Furthermore, these methods often struggle to produce good embeddings for less frequent word senses.

Methods

The core of this paper is a novel approach that leverages word sense definitions to learn word sense embeddings. The authors propose using Recurrent Neural Networks (RNNs), specifically mentioning the suitability of GRUs and LSTMs for capturing long-term dependencies within definitions.

The methodology involves several key steps:

  1. Definition Understanding Model: An RNN is trained to process a word sense definition word by word. The final hidden state of the RNN is then mapped to a sense embedding space. This model learns to capture the semantic meaning of a definition.
  2. Training with Monosemous Words: Initially, the RNN-based definition understanding model is trained using definitions of monosemous words (words with only one sense). The sense embeddings for these words are initialized with their pre-trained word embeddings (from a corpus like Skip-gram). The objective is to make the RNN-generated sense embedding similar to the initialized word embedding.
  3. Learning Word Sense Embeddings (Joint Training): This is the crucial step where definitions of multi-sense words are incorporated. The sense embeddings are initialized using word embeddings of synonyms or contextually similar words from definitions. The model then jointly trains the RNN-based definition understanding model and the word sense embeddings. This means the definition understanding model learns from the sense embeddings, and the sense embeddings are refined based on the definitions.
  4. Representing Words in Definitions with Sense Embeddings: In a final training phase, the authors perform Word Sense Disambiguation (WSD) on the words within the definitions themselves. They then use the learned sense embeddings to represent these words, further refining both the sense embeddings and the definition understanding model. This step treats the learned sense embeddings as a form of "understanding" natural language descriptions.

Datasets

  • WordNet 3.0: Used as the lexical ontology to acquire definitions for word senses.
  • Google News dataset (approx. 100B words): Pre-trained 300-dimensional word embeddings (from Skip-gram) were used for initialization and as a baseline.
  • WordSim-353 dataset: Used for quantitative evaluation on word similarity tasks.
  • Stanford's Contextual Word Similarities (SCWS) dataset: Used for evaluating sense embeddings in a contextual setting.
  • Semeval-2007 coarse-grained all-words WSD task: Used for evaluating the learned sense embeddings in a word sense disambiguation task.

Key Insights and Ideas

  • Definitions as a Rich Source: The paper demonstrates that dictionary definitions, despite their relatively small size compared to corpora, contain sufficient semantic information to learn high-quality word sense embeddings.
  • RNNs for Semantic Composition: RNNs are shown to be effective in capturing the compositional nature of definitions and mapping them to meaningful semantic representations.
  • Bridging Distributional and Compositional Semantics: By initializing with corpus-based word embeddings and then training on definitions, the approach effectively combines distributional semantics (from word embeddings) with compositional semantics (from definitions).
  • Handling Rare Senses: The definition-based approach naturally treats all senses equally, as definitions don't inherently favor common words, thus offering a potential advantage for learning embeddings of rare word senses.
  • Self-Supervised Learning from Definitions: The entire process can be viewed as a form of self-supervised learning, where the structure and meaning within definitions provide the supervision signal.

Key Results

  • Improved Word Similarity: The proposed method significantly outperforms baseline word embeddings on the WordSim-353 dataset. The improvements are more pronounced when using LSTM and GRU models compared to vanilla RNNs.
  • Effective Contextual Understanding: On the SCWS dataset, the system achieves competitive results, demonstrating that the learned sense embeddings can capture semantic nuances in context.
  • Strong WSD Performance: The learned sense embeddings achieve state-of-the-art or near state-of-the-art performance on the Semeval-2007 WSD task, outperforming previous methods that relied on corpus-based WSD.
  • Qualitative Evidence: Nearest neighbor analysis shows that word sense embeddings are associated with specific senses, unlike word embeddings which tend to represent the most common sense. The model also demonstrates an ability to match descriptions to relevant word senses.

Limitations and Future Work

  • Reliance on Lexical Ontologies: The approach is dependent on the availability and quality of lexical ontologies like WordNet for definitions.
  • Initialization Sensitivity: While the paper describes initialization strategies, the quality of initial embeddings for rare words or senses might still impact performance.
  • Comparison Fairness: The authors acknowledge that direct comparisons with other methods are not always perfectly fair due to differences in resources used.

Future work could involve exploring other RNN architectures, investigating different initialization strategies, and applying the learned embeddings to a wider range of NLP tasks. The authors also suggest that the definition understanding model itself has potential applications beyond learning sense embeddings.

Summary updated

Abstract

Word embeddings play a significant role in many modern NLP systems. Since learning one representation per word is problematic for polysemous words and homonymous words, researchers propose to use one embedding per word sense. Their approaches mainly train word sense embeddings on a corpus. In this paper, we propose to use word sense definitions to learn one embedding per word sense. Experimental results on word similarity tasks and a word sense disambiguation task show that word sense embeddings produced by our approach are of high quality.
BibTeX
@inproceedings{li2016learning,
  author = {Qi Li and Tianshi Li and Baobao Chang},
  title = {{Learning Word Sense Embeddings from Word Sense Definitions}},
  booktitle = {NLPCC/ICCPOL},
  year = {2016},
  doi = {10.1007/978-3-319-50496-4\_19},
}
Figure