Improving unsupervised vector-space thematic fit evaluation via role-filler prototype clustering

Clayton Greenberg, Asad B. Sayeed, Vera DembergNAACL 2015 · DOI

Introduction

In distributional semantics, thematic fit measures the plausibility of a noun filling a specific semantic role for a verb—for example, how well "hammer" fits as an instrument for the verb "hit" compared to "feather." Traditional unsupervised models evaluate this by creating a "prototype" vector, typically the average of the most frequent nouns appearing in that role. However, this "one-size-fits-all" centroid approach fails significantly when a verb is polysemous.

The Challenge of Polysemy

When a verb has multiple meanings, its typical role-fillers are distributionally diverse. Consider the verb serve and its "with"-prepositional objects. It may be used in culinary contexts (serve with salad), military contexts (serve with a regiment), or legal contexts (serve with a writ). Averaging these diverse vectors results in a prototype that represents a semantic "mixture" rather than any real-world usage, making it a poor benchmark for evaluating new candidate nouns.

Figure 1: Vector space representation of the verb 'serve' showing how clustering captures distinct semantic senses (food, military, legal) that a single centroid would obscure.

Methodology: Multi-Prototype Clustering

The authors propose moving from a single centroid to a multi-prototype approach called kClusters. This method utilizes Agglomerative Clustering on the top $n$ typical role-fillers retrieved from a distributional model.

Distributional Models

The study evaluates three distinct vector spaces:

  1. TypeDM: A standard model based on syntactic dependencies.
  2. SDDM: A model based on semantic role labeling (SRL).
  3. SDDMX: A novel extension of SDDM that incorporates links between nouns belonging to the same predicate instance.

The kClusters Approach

Instead of a fixed number of clusters, the model uses the Variance Ratio Criterion (VRC) to dynamically determine the optimal number of clusters (ranging from 3 to 10). To evaluate a new noun, the model calculates its cosine similarity to every cluster centroid and selects the maximum score. This logic effectively asks if the noun fits any of the verb's known senses, rather than measuring its similarity to an average of all senses.

Evaluation and Results

The models were tested against four gold-standard human judgment datasets: Padó (2007), McRae et al. (1998), and two Ferretti et al. (2001) datasets focusing on instruments and locations. Performance was measured using Spearman’s $\rho$ to determine the correlation between model scores and human plausibility ratings.

Table 1: Spearman's rho correlations between model scores and human judgments. The kClusters method (bolded) shows superior performance across multiple distributional models.

As shown in the results, the kClusters method consistently outperformed the standard single Centroid and OneBest (nearest neighbor) baselines across nearly all datasets. In the aggregate evaluation, kClusters using the TypeDM model achieved a correlation of 0.385, compared to the 0.354 achieved by the centroid baseline.

Discussion

The success of the clustering approach highlights two key insights. First, Maximum Similarity is a more robust signal than average similarity for ambiguous words. Second, clustering acts as a natural noise filter; by isolating non-prototypical role-fillers into their own clusters, the model maintains cleaner prototypes for the primary senses.

The authors also noted that roles like locations and instruments benefited more from clustering than agents and patients, suggesting that the "where" and "how" of actions are often more distributionally diverse than the "who" or "what."

Summary updated

BibTeX
@inproceedings{greenberg2015improving,
  author = {Clayton Greenberg and Asad B. Sayeed and Vera Demberg},
  title = {{Improving unsupervised vector-space thematic fit evaluation via role-filler prototype clustering}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
  doi = {10.3115/v1/n15-1003},
}
Figure