Dynamic and Static Prototype Vectors for Semantic Composition

Siva Reddy, Ioannis P. Klapaftis, Diana McCarthy, Suresh ManandharIJCNLP 2011

Introduction: The Conflation Problem

In distributional semantics, words are typically represented as high-dimensional vectors. While effective for individual words, this approach faces a significant hurdle during semantic composition—the process of combining word meanings into phrases. The primary issue is conflation: a single vector for a polysemous word like "hunting" must represent all its senses simultaneously, from searching for a home to tracking animals.

When composing a phrase like "house hunting," the standard additive or multiplicative models retain "ghost" features from irrelevant senses. For instance, the vector for "hunting" contains strong associations with animal and kill, which introduce noise into the final representation of a real estate search. This paper explores methods to filter these irrelevant features to create cleaner, more accurate semantic compositions.

Table 1: Example of conflated vectors for 'house' and 'hunting', showing high values for irrelevant dimensions like 'animal' and 'kill' for the word 'hunting'.

Methodology

The authors evaluate three distinct strategies for representing word senses during composition:

1. Static Single Prototype (Baseline)

This is the standard approach where every occurrence of a word contributes to a single, global vector. It serves as the baseline but suffers most acutely from the conflation problem described above.

2. Static Multi-Prototypes

This method uses Word Sense Induction (WSI) to pre-cluster word occurrences into distinct sense vectors before composition. Using the Chinese Whispers graph-based clustering algorithm, the model identifies clusters of related terms (collocations) to define discrete senses. For a compound like "house hunting," the model selects the pair of pre-defined senses (one from "house," one from "hunting") that are most similar to each other.

Figure 1: The process of inducing senses for the target word 'mouse' using graph-based clustering of collocations.

3. Dynamic Prototypes (The Proposed Approach)

Rather than relying on rigid, pre-defined clusters, the Dynamic Prototype approach builds a custom vector for a word on the fly, specifically tailored to its partner in a compound. To represent "hunting" in the context of "house," the model ranks thousands of individual instances (exemplars) of "hunting" from the ukWaC corpus based on their proximity to house-related collocations. A "Dynamic Prototype" is then created by averaging only the top $n\%$ of these relevant instances.

Experimental Results

The models were evaluated against the Mitchell and Lapata (2010) dataset, which contains 108 noun-noun compounds rated for similarity by human annotators. Performance was measured using Spearman’s Correlation to determine how well model rankings matched human intuition.

Model Spearman’s Correlation
Static Multi-Prototypes 0.4150
Compound Prototype (Direct Lookup) 0.4152
Static Single Prototype (Baseline) 0.6104
Dynamic Prototypes (Top 2%) 0.6552

The results demonstrate that the Dynamic Prototype approach significantly outperforms the standard baseline. Interestingly, the Static Multi-Prototype method performed poorly, suggesting that forcing language into rigid, unsupervised clusters can be counterproductive compared to fluid, context-driven filtering.

Key Insights and Discussion

  • Context as a Filter: The success of the dynamic method suggests that context acts as a semantic filter rather than just a label. By activating only the most relevant "memories" (exemplars) of a word, the model effectively eliminates noise.
  • The Sparsity of Relevance: The best performance was achieved using only the top 2% of relevant exemplars. Including more data (up to 80%) actually degraded performance as irrelevant senses began to leak back into the vector.
  • Limitations: While powerful, the dynamic approach is computationally expensive, requiring the ranking of thousands of exemplars for every new composition. Additionally, the current study is limited to noun-noun compounds; future work is needed to extend this to more complex syntactic structures.

Summary updated

BibTeX
@inproceedings{reddy2011dynamic,
  author = {Siva Reddy and Ioannis P. Klapaftis and Diana McCarthy and Suresh Manandhar},
  title = {{Dynamic and Static Prototype Vectors for Semantic Composition}},
  booktitle = {IJCNLP},
  year = {2011},
}
Figure