Introduction
In natural language processing, word meanings are typically represented as high-dimensional vectors. However, standard "single-prototype" models suffer from conflation, where all meanings of a word (e.g., "bank" as a financial institution vs. "bank" as a river edge) are compressed into a single vector.
While "multi-prototype" models address this by clustering word occurrences into distinct senses, they often fail to capture the nuances of polysemy. Unlike homonyms (unrelated meanings), polysemous words like "line" or "run" have many senses that share a common metaphoric core. Standard clustering treats these senses as entirely independent, ignoring the shared structure that defines the word's general character. This paper introduces tiered clustering, a probabilistic mixture model that represents word meanings by separating "core" background features from "sense-specific" variations.
Methodology
The authors propose a tiered clustering approach that performs a form of "soft feature selection" during the clustering process. The model architecture consists of two primary components:
- A Background Model: A single component that accounts for features shared across all occurrences of a word (context-independent features).
- A Dirichlet Process Mixture Model (DPMM): An infinite capacity clustering model that automatically determines the number of "senses" or "prototypes" for a word based on the data.
For every feature in a word's context, the model decides whether it belongs to the background (common to all senses) or to a specific cluster (unique to one sense). Inference is handled via an efficient collapsed Gibbs sampler. Semantic relatedness is measured using AvgSim, which calculates the average cosine similarity between all pairs of prototypes for two words.
Evaluation and Datasets
The model was trained on a snapshot of English Wikipedia (Sept. 2009), comprising 2.8 million articles. Performance was evaluated across several benchmarks:
- WordSim-353: A standard dataset for word pair similarity.
- WN-Evocation: A large-scale dataset reflecting real-world lexical sparsity.
- Padó Dataset: Used to evaluate selectional preference in verb-noun pairs.
Results and Discussion
By siphoning off common background noise into the background tier, the resulting sense clusters become semantically purer. The model excels particularly when words have high degrees of shared structure. For instance, in the word "radio," the background model captures general terms like "station" and "broadcast," while specific clusters isolate technical vs. entertainment contexts.
On the WordSim-353 "high polysemy" subset, tiered clustering achieved a Spearman correlation ($\rho$) of 0.831, significantly outperforming standard multi-prototype (0.793) and single-prototype (0.760) models. The model's strongest performance was in selectional preference on the Padó dataset, reaching $\rho = 0.294$. This highlights that for verbs, capturing the shared argument structure is more critical than splitting arguments into disjoint clusters.
Conclusion
Tiered clustering effectively addresses the conflation and polysemy issues in lexical semantics. While the current model uses simple unigram context windows, future work could incorporate more expressive features like dependency parses. Additionally, the authors suggest extending the model to include multiple background tiers to account for commonalities between different arguments of a verb, moving toward a more complex hierarchical structure.
Summary updated