Investigating Word-Class Distributions in Word Vector Spaces

Ryohei Sasano, Anna KorhonenACL 2020 · DOI

Introduction

While word embeddings have revolutionized Natural Language Processing by capturing semantic relationships, the internal geometry of specific "word classes"—such as "types of mammals" or "objects of the verb play"—remains poorly understood. Traditional approaches often rely on simple centroid-based models, assuming classes form spherical clusters. This research investigates whether more complex geometric or discriminative models better capture the distribution of these classes and their "typicality" as perceived by humans.

Modeling Word-Class Geometry

The study evaluates eight modeling approaches to define word-class boundaries and membership degrees. These are categorized into three groups:

  1. Proximity-based: Includes the standard Centroid (CENT) baseline and k-Nearest Neighbor (kNN), which focuses on local density.
  2. Geometry-based: These models account for the "shape" of the distribution. The Gaussian Model (GM) treats a class as an elliptical cloud, while the Gaussian Mixture Model (GMM) allows for multiple distinct sub-clusters.
  3. Discriminative: These leverage both positive and negative instances. Notable among these is the OffSet model, which calculates the vector difference between positive and negative centroids, and Support Vector Machines (SVMs) with linear and non-linear kernels.
Comparison of modeling approaches: CENT, GM, GMM, and OffSet.

Experimental Setup

The models were tested across English and Japanese datasets, focusing on three tasks:

  • Selectional Preference (SP): Identifying valid direct objects for 1,000 verbs extracted from Google Books N-grams.
  • WordNet Hierarchy: Modeling sets based on synsets and hyponyms (e.g., "placental mammals").
  • Typicality: Correlating model outputs with human ratings from the Rosch (1975) dataset, which measures how "prototypical" a word is for its category (e.g., a "chair" vs. a "stove" for the category "furniture").

Key Insights and Results

The experiments yielded several critical findings regarding the nature of word-class distributions:

The Necessity of Negative Examples

Models utilizing negative instances consistently outperformed those relying solely on positive data. This suggests that defining a semantic category requires understanding not just what belongs, but what is excluded. In the Selectional Preference task, the non-linear SVM ($SVM_R$) achieved the highest Mean Reciprocal Rank (MRR), often doubling the performance of the simple centroid baseline (e.g., 0.4922 vs 0.2600 in Japanese datasets).

Visualization of positive and negative instances in vector space with respective centroids.

The Typicality Paradox

A significant finding emerged regarding human intuition: while non-linear SVMs were superior at identifying class boundaries (membership classification), they were less effective at capturing degrees of membership. For the Typicality task, the OffSet model achieved the highest Spearman correlation with human ratings ($\rho = 0.3516$), outperforming the high-accuracy $SVM_R$ (0.2795). This suggests that the "direction" away from non-members is a primary indicator of prototypicality.

Geometric Stretching

The transition from CENT to GM showed that word classes are not perfectly spherical; they are "stretched" along specific semantic dimensions. Modeling this variance improves performance, proving that geometry matters in semantic representation.

Conclusion and Future Work

This investigation demonstrates that simple centroids are insufficient for modeling word-class distributions. Discriminative learning and geometric modeling provide a much more accurate representation of semantic space. Future work will look to extend these findings to contextual embeddings (like BERT) to address polysemy and investigate the impact of word frequency on spatial positioning.

Summary updated

BibTeX
@inproceedings{sasano2020investigating,
  author = {Ryohei Sasano and Anna Korhonen},
  title = {{Investigating Word-Class Distributions in Word Vector Spaces}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.337},
}
Figure