Modeling Word Meaning in Context with Substitute Vectors

Oren Melamud, Ido Dagan, Jacob GoldbergerNAACL 2015

TL;DR

This paper introduces a novel approach to modeling word meaning in context by representing contexts as "substitute vectors" – collections of words that could plausibly fill a target word's slot. This method significantly outperforms existing techniques on lexical substitution tasks, demonstrating a more nuanced understanding of word meaning in specific situations.

What Problem Does This Paper Address?

Understanding the meaning of a word is tricky because words can have multiple meanings (polysemy) and their meaning often depends heavily on the surrounding words (context). Traditional distributional models represent a word's meaning as an aggregation of its neighboring words (a "bag-of-words" or BOW). While this captures some information, it struggles with polysemy and doesn't fully leverage the subtle relationships between words in a context. More recent approaches try to bias a word's general meaning towards its specific context, but the way contexts are represented remains a key challenge. This paper tackles this by proposing a new way to represent contexts that is more informative for capturing word meaning in context.

Methods

The core innovation lies in the concept of substitute vectors. Instead of just listing neighboring words, a substitute vector represents a context by listing potential "fillers" for the target word's slot, weighted by how well they fit. This is inspired by earlier work but refined here.

The authors propose two main contributions:

  1. A refined substitute vector similarity measure: They use Positive Pointwise Mutual Information (PPMI) to weight the fillers in the substitute vector. This measure is found to be particularly effective for comparing contexts.
  2. A novel model for word meaning in context: This model uses the refined substitute vectors to represent contexts.
  3. Out-of-context representation: The general meaning of a word type is an average of the substitute vectors of all its observed contexts.
  4. In-context representation: The meaning of a specific word instance is a weighted average of its contexts, where contexts more similar to the given one are given higher weight. This "paraphrase vector" is essentially a context-sensitive meaning.

They also introduce a pseudo-word approach to generate large-scale, sense-annotated datasets for evaluation, which is crucial for tasks like lexical substitution.

Datasets

  • ukWaC: A large web corpus used for learning the language models and word representations.
  • LS07 (SemEval 2007 Lexical Substitution Dataset): A benchmark dataset for evaluating lexical substitution, containing 201 target words and ~2,010 instances.
  • LS14 (Kremer et al., 2014): A larger, more recent dataset with over 15,000 instances, derived from news and fiction corpora.

The authors also created their own pseudo-word dataset for evaluating context similarity measures, which is publicly available.

Key Insights and Ideas

  • Context is king, but how we represent it matters: The paper strongly advocates for richer context representations beyond simple bag-of-words. Substitute vectors, by focusing on potential fillers, capture more nuanced contextual information.
  • Similarity is key for context-sensitive meaning: The effectiveness of their in-context model hinges on accurately measuring context similarity. Their proposed PPMI-based substitute vector similarity measure proves superior to traditional methods.
  • Polysemy is best handled by context-specific representations: The "paraphrase vectors" (in-context representations) show significant improvements over out-of-context representations, especially for ambiguous words.
  • Unsupervised learning can achieve state-of-the-art: The model learns entirely from plain text, demonstrating the power of unsupervised methods for complex NLP tasks.
  • Ranking vs. Prediction: The paper evaluates their model on both predicting paraphrases and ranking them, showing strong performance across both.

Key Results

  • Context Similarity: Their substitute vector similarity measure (SUBppmi) significantly outperforms BOW and CBOW baselines on precision at top-1 and top-1% for retrieving similar contexts.
  • Lexical Substitution (Prediction): The proposed in-context model (Pin) significantly outperforms state-of-the-art baselines on both LS07 and LS14 datasets for predicting paraphrases. For example, on LS07, their best model achieved a score of 52.03 in the "best-m" subtask, compared to ~37 for previous methods.
  • Lexical Substitution (Ranking): The model also achieves top results in ranking paraphrases, outperforming previous unsupervised methods and achieving scores comparable to supervised methods on LS07. Their best result on LS07 (test-set) for GAP was 55.2, and on LS14 it was 50.2.
  • Efficiency: The model is computationally efficient, with generation times comparable to existing methods. They also show that clustering substitute vectors can reduce memory consumption significantly with only a minor performance drop.

Limitations and Future Work

  • Quality of Substitute Vectors: The authors acknowledge that the quality of the generated substitute vectors can limit performance. They suggest using better language models, such as neural language models, for generating these vectors in the future.
  • Memory Consumption: While addressed by clustering, the "online" mode of the model can still require substantial memory for large vocabularies.
  • Ambiguity in LS14: The smaller performance gains on LS14 compared to LS07 might be due to LS14's target words being less ambiguous by design.
  • Future Directions: The authors suggest exploring their substitute vector approach for other tasks like part-of-speech induction and word sense induction, which have been addressed using similar techniques but with different embedding strategies.

Summary updated

BibTeX
@inproceedings{melamud2015modeling,
  author = {Oren Melamud and Ido Dagan and Jacob Goldberger},
  title = {{Modeling Word Meaning in Context with Substitute Vectors}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
}
Figure