TL;DR
This paper introduces a novel "multi-prototype" approach to representing word meaning in vector spaces. Instead of a single vector per word, it uses multiple "sense-specific" vectors derived from clustering word occurrences, significantly improving semantic similarity judgments and near-synonym prediction compared to traditional single-prototype and exemplar-based methods.
What Problem Does This Paper Address?
Traditional vector-space models of word meaning represent each word with a single vector. This approach struggles with lexical ambiguity, specifically homonymy (words with distinct meanings, like "bat" for an animal and a sports equipment) and polysemy (words with related meanings, like "bank" for a financial institution and a river edge). Because a single vector cannot capture these multiple meanings, these models often fail to accurately represent semantic similarity. For instance, the word "club" might be considered similar to both "bat" and "association," even though "bat" and "association" are not semantically related. This violation of the "triangle inequality" in word meaning poses a fundamental challenge for single-vector representations. Furthermore, most existing models are context-independent, meaning they don't adapt word meaning based on how the word is used in a sentence.
Methods
The core innovation is the "multi-prototype" model. For each word, the authors collect all its occurrences in a corpus and represent each occurrence as a feature vector. These occurrence vectors are then clustered using an unsupervised word sense discovery technique (specifically, a mixture of von Mises-Fisher distributions, movMF). Each cluster's centroid becomes a "sense-specific" prototype vector for the word.
To measure the similarity between two words (w and w'), they propose two metrics:
- AvgSim: The average similarity across all pairs of prototype vectors from the two words. This captures overall semantic overlap.
- MaxSim: The maximum similarity between any pair of prototype vectors. This is sensitive to even a single shared sense.
For contextual similarity, they introduce AvgSimC and MaxSimC, which weight prototype similarities by the likelihood of the context belonging to a particular cluster. This allows the model to select the most appropriate sense-specific prototype for a given context.
Datasets
- Training/Evaluation Corpora:
- Wikipedia Snapshot (Sept. 29th, 2009): Approximately 2.8 million articles, totaling 2.05 billion words.
- English Gigaword Corpus (3rd Edition): Approximately 6.6 million articles, totaling 3.9 billion words.
- Evaluation Benchmark:
- WordSim-353: A dataset containing 353 word pairs with human similarity judgments on a 1-10 scale, covering both monosemous and polysemous words.
- Usage Similarity (USim) dataset (from Erk et al., 2009): Used to evaluate how well prototype vectors correspond to human intuition for specific word usages.
- Near-Synonym Prediction Task: A custom evaluation where human subjects on Amazon Mechanical Turk chose between synonyms predicted by the single-prototype model and the multi-prototype model, both in isolation and in context.
Key Insights and Ideas
- Lexical Ambiguity is Key: The paper strongly argues that single vectors are insufficient for representing word meaning due to homonymy and polysemy. Capturing multiple senses is crucial for accurate semantic modeling.
- Clustering as Sense Discovery: Unsupervised clustering of word occurrences is presented as a powerful mechanism for discovering "sense-specific" prototypes without relying on pre-defined lexicons like WordNet.
- Modular Design: The multi-prototype framework is modular, allowing integration with various clustering methods and feature representations (like tf-idf and x²).
- Psychological Plausibility: The authors find that the variance in their prototype similarities correlates with human annotator disagreement, suggesting a degree of psychological plausibility in how their model captures uncertainty in word meaning.
- Context Matters: The contextualized version of their model (AvgSimC) shows significant improvements, especially for minority senses, demonstrating the importance of context-aware word representations.
Key Results
- WordSim-353 Performance: The multi-prototype approach (AvgSim) consistently outperformed the single-prototype baseline across all corpora and feature types, achieving state-of-the-art results with pruned tf-idf features. For example, with Wikipedia and tf-idf, AvgSim achieved a Spearman correlation of 0.76±0.01 for K=20 clusters, compared to 0.53±0.02 for the single-prototype.
- Near-Synonym Prediction: The multi-prototype model was preferred by human evaluators significantly more often than the single-prototype model, particularly for homonymous words and when predicting synonyms for minority senses in context.
- Exemplar vs. Multi-Prototype: The multi-prototype approach performed comparably to or better than a pure exemplar approach, while offering computational advantages (O(K²) comparisons vs. O(n²)).
- Impact of K: Performance was sensitive to the number of clusters (K). While smaller K values generally yielded better results for AvgSim, combining prototypes from multiple clusterings (2-5, 10, 20, 50, 100) provided robust performance across different K values.
Limitations and Future Work
- Sensitivity to K: The performance is sensitive to the choice of K (number of clusters). While combining prototypes helps, finding the optimal K for each word remains a challenge.
- Noise in High-Cluster Counts: With a large number of clusters, individual prototypes can become noisy, especially if they represent rare outlier occurrences. This affects tasks like near-synonym prediction more than general similarity.
- Prototype-Sense Mismatch: The authors found that while prototypes capture useful semantic distinctions, they don't necessarily align with human-defined word senses (e.g., the "hurricane" sense of "position").
- Future Work:
- Feature Representations: Exploring more powerful feature representations like Latent Semantic Analysis (LSA) or Explicit Semantic Analysis (ESA).
- Nonparametric Clustering: Using models with infinite capacity (like Dirichlet Process Mixture Models) to allow the number of prototypes to vary per word, better handling polysemy.
- Joint Modeling: Developing a joint model where senses for different words influence each other, potentially improving performance for rarer words.
- Comparison to Lexicons: Comparing learned prototypes to those derived from supervised sense inventories (like WordNet).
Summary updated