Abstract
Traditional computational models of lexical semantics often rely on a single, global similarity metric to represent word relationships. However, human language is inherently multi-faceted, with words relating through diverse thematic, syntactic, and paradigmatic lenses. We present the Multi-View Mixture (MVM) model, a probabilistic approach that captures these "cross-cutting" relationships by allowing words to belong to multiple overlapping clusterings simultaneously.
The Challenge of Single-Metric Similarity
In computational linguistics, words are typically represented as vectors in a high-dimensional space where proximity denotes similarity. Yet, human word similarity is context-dependent and multi-dimensional. For instance, wine and bottle share a thematic (syntagmatic) relationship, while wine and vinegar share a paradigmatic relationship as types of liquids.
Forcing these distinct types of relationships into a single metric leads to "metric violations," such as the failure of the triangle inequality. The fundamental challenge is developing a model that can learn which subsets of features—or "subspaces"—should trigger specific interpretations of similarity.
The Multi-View Mixture (MVM) Model
The Multi-View Mixture (MVM) model addresses this by combining Latent Dirichlet Allocation (LDA) with Dirichlet Process Mixture Models (DPMMs).
- Soft Feature Assignment (LDA): Instead of assigning a context feature to a single view, MVM uses LDA to perform "soft" assignments. This allows a single feature (e.g., "___ for sale") to contribute to multiple perspectives of similarity.
- Flexible Clustering (DPMM): Within each view, a DPMM clusters words. The use of a Dirichlet Process allows the model to automatically determine the optimal number of clusters for each specific view, enhancing scalability and nuance.
This architecture allows the model to isolate noise effectively. In traditional single-view models, rare or noisy features can degrade the entire clustering. MVM shunts these into separate views, preserving the coherence of primary semantic clusters and maintaining high interpretability even at high levels of granularity.
Evaluation and Results
The model was evaluated using "Set Intrusion" tasks on massive datasets, including Google Web n-grams and Wikipedia. Human raters were tasked with identifying "intruder" words or contexts within model-generated clusters. High human accuracy in these tasks indicates that the model's clusters are intuitive and coherent.
MVM consistently outperformed standard LDA and DPMM baselines across various tasks. On context intrusion tasks, MVM achieved 64.5% accuracy, compared to 57.4% for DPMM and 49.5% for LDA. When incorporating Wikipedia data for broader thematic coverage, MVM reached 74.9% accuracy in word intrusion tasks. Furthermore, MVM showed significantly higher inter-annotator agreement (Fleiss’ kappa of 0.44–0.49), suggesting its clusters are more "obviously" correct than those of traditional models.
Conclusion and Future Directions
The MVM model provides a scalable, probabilistic framework for capturing the cross-cutting nature of lexical semantics. By moving beyond a single similarity metric, it offers a more human-like understanding of word relationships. Future work includes extending the model to handle hierarchical relationships and applying its noise-isolation properties to other domains, such as computer vision, where identifying objects in cluttered scenes requires ignoring irrelevant features.
Summary updated