Abstract
The computation of meaning similarity as operationalized by vector-based models has found widespread use in many tasks ranging from the acquisition of synonyms and paraphrases to word sense disambiguation and textual entailment. Vector-based models are typically directed at representing words in isolation and thus best suited for measuring similarity out of context. In this paper, we propose a probabilistic framework for measuring similarity in context. Central to our approach is the intuition that word meaning is represented as a probability distribution over a set of latent senses and is modulated by context. Experimental results on lexical substitution and word similarity show that our algorithm outperforms previously proposed models.
The Challenge of Polysemy
Traditional Natural Language Processing (NLP) often relies on the "distributional hypothesis," which suggests that words appearing in similar contexts share similar meanings. This led to the rise of vector-space models (VSMs), where each word is represented as a single point in a high-dimensional space.
The glaring problem with this approach is polysemy. A static vector averages all possible meanings of a word, creating a "blurred" representation that doesn't truly fit any specific context. For example, the word still means something very different in the phrase "a still day" versus "still images."
While some researchers have attempted to fix this using vector arithmetic (like adding or multiplying vectors), these methods often lack a solid theoretical foundation for how context actually modifies meaning.
A Probabilistic Framework for Contextualization
Dinu and Lapata propose a framework where a word’s meaning is not a fixed point, but a probability distribution over a set of global "latent senses."
- Latent Senses: Instead of raw co-occurrence counts, they represent a word $t_i$ as a vector of probabilities $P(z_k | t_i)$, where $z_k$ represents a hidden sense or topic.
- Contextualization: When a word appears with a context feature $c_j$, the model updates the distribution. The "contextualized" meaning is $P(z_k | t_i, c_j)$. Essentially, the context acts as a filter that shifts the probability mass toward the senses that make the most sense in that specific situation.
- Engines: To identify these latent senses, the authors tested Non-negative Matrix Factorization (NMF) and Latent Dirichlet Allocation (LDA).
- Similarity Measures: They found that the scalar product worked best for measuring the closeness of two distributions because it naturally rewards distributions that are both similar and "focused" (less ambiguous).
Experimental Results
The models were trained on the 1.7-billion-word GigaWord collection and evaluated on both out-of-context (WordSim-353) and in-context (SemEval 2007 Lexical Substitution) benchmarks.
Lexical Substitution
The contextualized probabilistic models (Cont-NMF and Cont-LDA) significantly outperformed traditional additive and multiplicative baselines. The best-performing model was a mixture of NMF models (Cont-NMF$_{MIX}$), which achieved a Kendall’s $\tau_b$ of 16.01, a significant jump over the simple vector space baseline of 11.05.
Word Similarity and Part-of-Speech Gains
On the WordSim-353 dataset, LDA achieved a Spearman correlation of $\rho = 53.39$, outperforming the LSA baseline of $49.43$. Furthermore, the model showed the most significant improvements on verbs and nouns, which are often the most polysemous and context-dependent parts of speech.
Conclusion and Future Work
By treating context as a modulator that reduces the entropy of a word's meaning, this framework provides a more robust way to handle polysemy than static vectors. While the current model relies on a conditional independence assumption between the target word and its context, future work may explore asymmetric models or more principled ways to aggregate information from an entire sentence rather than individual context words.
Summary updated