Measuring Distributional Similarity in Context

Georgiana Dinu, Mirella LapataEMNLP 2010 · OpenReview

Abstract

The computation of meaning similarity as operationalized by vector-based models has found widespread use in many tasks ranging from the acquisition of synonyms and paraphrases to word sense disambiguation and textual entailment. Vector-based models are typically directed at representing words in isolation and thus best suited for measuring similarity out of context. In this paper, we propose a probabilistic framework for measuring similarity in context. Central to our approach is the intuition that word meaning is represented as a probability distribution over a set of latent senses and is modulated by context. Experimental results on lexical substitution and word similarity show that our algorithm outperforms previously proposed models.

The Challenge of Polysemy

Traditional Natural Language Processing (NLP) often relies on the "distributional hypothesis," which suggests that words appearing in similar contexts share similar meanings. This led to the rise of vector-space models (VSMs), where each word is represented as a single point in a high-dimensional space.

The glaring problem with this approach is polysemy. A static vector averages all possible meanings of a word, creating a "blurred" representation that doesn't truly fit any specific context. For example, the word still means something very different in the phrase "a still day" versus "still images."

Lexical substitution examples for the polysemous word 'still' in different contexts.

While some researchers have attempted to fix this using vector arithmetic (like adding or multiplying vectors), these methods often lack a solid theoretical foundation for how context actually modifies meaning.

A Probabilistic Framework for Contextualization

Dinu and Lapata propose a framework where a word’s meaning is not a fixed point, but a probability distribution over a set of global "latent senses."

  1. Latent Senses: Instead of raw co-occurrence counts, they represent a word $t_i$ as a vector of probabilities $P(z_k | t_i)$, where $z_k$ represents a hidden sense or topic.
  2. Contextualization: When a word appears with a context feature $c_j$, the model updates the distribution. The "contextualized" meaning is $P(z_k | t_i, c_j)$. Essentially, the context acts as a filter that shifts the probability mass toward the senses that make the most sense in that specific situation.
  3. Engines: To identify these latent senses, the authors tested Non-negative Matrix Factorization (NMF) and Latent Dirichlet Allocation (LDA).
  4. Similarity Measures: They found that the scalar product worked best for measuring the closeness of two distributions because it naturally rewards distributions that are both similar and "focused" (less ambiguous).

Experimental Results

The models were trained on the 1.7-billion-word GigaWord collection and evaluated on both out-of-context (WordSim-353) and in-context (SemEval 2007 Lexical Substitution) benchmarks.

Lexical Substitution

The contextualized probabilistic models (Cont-NMF and Cont-LDA) significantly outperformed traditional additive and multiplicative baselines. The best-performing model was a mixture of NMF models (Cont-NMF$_{MIX}$), which achieved a Kendall’s $\tau_b$ of 16.01, a significant jump over the simple vector space baseline of 11.05.

Comparison of model performance on the lexical substitution task using Kendall’s $\tau_b$.

Word Similarity and Part-of-Speech Gains

On the WordSim-353 dataset, LDA achieved a Spearman correlation of $\rho = 53.39$, outperforming the LSA baseline of $49.43$. Furthermore, the model showed the most significant improvements on verbs and nouns, which are often the most polysemous and context-dependent parts of speech.

Conclusion and Future Work

By treating context as a modulator that reduces the entropy of a word's meaning, this framework provides a more robust way to handle polysemy than static vectors. While the current model relies on a conditional independence assumption between the target word and its context, future work may explore asymmetric models or more principled ways to aggregate information from an entire sentence rather than individual context words.

Summary updated

Abstract

The computation of meaning similarity as operationalized by vector-based models has found widespread use in many tasks ranging from the acquisition of synonyms and paraphrases to word sense disambiguation and textual entailment. Vector-based models are typically directed at representing words in isolation and thus best suited for measuring similarity out of context. In his paper we propose a probabilistic framework for measuring similarity in context. Central to our approach is the intuition that word meaning is represented as a probability distribution over a set of latent senses and is modulated by context. Experimental results on lexical substitution and word similarity show that our algorithm outperforms previously proposed models.
BibTeX
@inproceedings{dinu2010measuring,
  author = {Georgiana Dinu and Mirella Lapata},
  title = {{Measuring Distributional Similarity in Context}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2010},
  url = {https://openreview.net/forum?id=BJN9KMf_-r},
}
Figure