Introduction: The Conflation Problem
In distributional semantics, words are typically represented as high-dimensional vectors. While effective for individual words, this approach faces a significant hurdle during semantic composition—the process of combining word meanings into phrases. The primary issue is conflation: a single vector for a polysemous word like "hunting" must represent all its senses simultaneously, from searching for a home to tracking animals.
When composing a phrase like "house hunting," the standard additive or multiplicative models retain "ghost" features from irrelevant senses. For instance, the vector for "hunting" contains strong associations with animal and kill, which introduce noise into the final representation of a real estate search. This paper explores methods to filter these irrelevant features to create cleaner, more accurate semantic compositions.
Methodology
The authors evaluate three distinct strategies for representing word senses during composition:
1. Static Single Prototype (Baseline)
This is the standard approach where every occurrence of a word contributes to a single, global vector. It serves as the baseline but suffers most acutely from the conflation problem described above.
2. Static Multi-Prototypes
This method uses Word Sense Induction (WSI) to pre-cluster word occurrences into distinct sense vectors before composition. Using the Chinese Whispers graph-based clustering algorithm, the model identifies clusters of related terms (collocations) to define discrete senses. For a compound like "house hunting," the model selects the pair of pre-defined senses (one from "house," one from "hunting") that are most similar to each other.
3. Dynamic Prototypes (The Proposed Approach)
Rather than relying on rigid, pre-defined clusters, the Dynamic Prototype approach builds a custom vector for a word on the fly, specifically tailored to its partner in a compound. To represent "hunting" in the context of "house," the model ranks thousands of individual instances (exemplars) of "hunting" from the ukWaC corpus based on their proximity to house-related collocations. A "Dynamic Prototype" is then created by averaging only the top $n\%$ of these relevant instances.
Experimental Results
The models were evaluated against the Mitchell and Lapata (2010) dataset, which contains 108 noun-noun compounds rated for similarity by human annotators. Performance was measured using Spearman’s Correlation to determine how well model rankings matched human intuition.
| Model | Spearman’s Correlation |
|---|---|
| Static Multi-Prototypes | 0.4150 |
| Compound Prototype (Direct Lookup) | 0.4152 |
| Static Single Prototype (Baseline) | 0.6104 |
| Dynamic Prototypes (Top 2%) | 0.6552 |
The results demonstrate that the Dynamic Prototype approach significantly outperforms the standard baseline. Interestingly, the Static Multi-Prototype method performed poorly, suggesting that forcing language into rigid, unsupervised clusters can be counterproductive compared to fluid, context-driven filtering.
Key Insights and Discussion
- Context as a Filter: The success of the dynamic method suggests that context acts as a semantic filter rather than just a label. By activating only the most relevant "memories" (exemplars) of a word, the model effectively eliminates noise.
- The Sparsity of Relevance: The best performance was achieved using only the top 2% of relevant exemplars. Including more data (up to 80%) actually degraded performance as irrelevant senses began to leak back into the vector.
- Limitations: While powerful, the dynamic approach is computationally expensive, requiring the ranking of thousands of exemplars for every new composition. Additionally, the current study is limited to noun-noun compounds; future work is needed to extend this to more complex syntactic structures.
Summary updated