Introduction
While word embeddings have revolutionized Natural Language Processing by capturing semantic relationships, the internal geometry of specific "word classes"—such as "types of mammals" or "objects of the verb play"—remains poorly understood. Traditional approaches often rely on simple centroid-based models, assuming classes form spherical clusters. This research investigates whether more complex geometric or discriminative models better capture the distribution of these classes and their "typicality" as perceived by humans.
Modeling Word-Class Geometry
The study evaluates eight modeling approaches to define word-class boundaries and membership degrees. These are categorized into three groups:
- Proximity-based: Includes the standard Centroid (CENT) baseline and k-Nearest Neighbor (kNN), which focuses on local density.
- Geometry-based: These models account for the "shape" of the distribution. The Gaussian Model (GM) treats a class as an elliptical cloud, while the Gaussian Mixture Model (GMM) allows for multiple distinct sub-clusters.
- Discriminative: These leverage both positive and negative instances. Notable among these is the OffSet model, which calculates the vector difference between positive and negative centroids, and Support Vector Machines (SVMs) with linear and non-linear kernels.
Experimental Setup
The models were tested across English and Japanese datasets, focusing on three tasks:
- Selectional Preference (SP): Identifying valid direct objects for 1,000 verbs extracted from Google Books N-grams.
- WordNet Hierarchy: Modeling sets based on synsets and hyponyms (e.g., "placental mammals").
- Typicality: Correlating model outputs with human ratings from the Rosch (1975) dataset, which measures how "prototypical" a word is for its category (e.g., a "chair" vs. a "stove" for the category "furniture").
Key Insights and Results
The experiments yielded several critical findings regarding the nature of word-class distributions:
The Necessity of Negative Examples
Models utilizing negative instances consistently outperformed those relying solely on positive data. This suggests that defining a semantic category requires understanding not just what belongs, but what is excluded. In the Selectional Preference task, the non-linear SVM ($SVM_R$) achieved the highest Mean Reciprocal Rank (MRR), often doubling the performance of the simple centroid baseline (e.g., 0.4922 vs 0.2600 in Japanese datasets).
The Typicality Paradox
A significant finding emerged regarding human intuition: while non-linear SVMs were superior at identifying class boundaries (membership classification), they were less effective at capturing degrees of membership. For the Typicality task, the OffSet model achieved the highest Spearman correlation with human ratings ($\rho = 0.3516$), outperforming the high-accuracy $SVM_R$ (0.2795). This suggests that the "direction" away from non-members is a primary indicator of prototypicality.
Geometric Stretching
The transition from CENT to GM showed that word classes are not perfectly spherical; they are "stretched" along specific semantic dimensions. Modeling this variance improves performance, proving that geometry matters in semantic representation.
Conclusion and Future Work
This investigation demonstrates that simple centroids are insufficient for modeling word-class distributions. Discriminative learning and geometric modeling provide a much more accurate representation of semantic space. Future work will look to extend these findings to contextual embeddings (like BERT) to address polysemy and investigate the impact of word frequency on spatial positioning.
Summary updated