TL;DR
This paper explores the "hidden" geometry of dependency-conditioned word embeddings—representations that treat a word and its syntactic role (e.g., "bank" as a direct object) as a single unit. The authors discover that these embeddings are globally organized by the part-of-speech of their context, while locally, they act as powerful filters that separate a word's different meanings based on its grammatical usage.
What Problem Does This Paper Address?
Standard distributional semantic models (like Word2Vec or GloVe) typically produce a single vector for each lemma, effectively squashing all of a word's different meanings and syntactic functions into one point in space. While dependency-based models use syntax to inform these vectors, the actual lemma-relation pairs (e.g., "bank" acting as a compound noun vs. "bank" as an oblique object) are rarely studied as a standalone semantic space.
The authors set out to map this space. They want to know: How is the geometry of these syntactic units organized? Does the structure depend on the learning algorithm (count-based vs. predictive)? And can these representations naturally distinguish between different senses of polysemous words?
Methods
The researchers trained four different representation models on the WikiText-103 corpus:
- PPMI-5000: A sparse, count-based model using the 5,000 most frequent contexts.
- PPMI-SVD: A dense version of the count-based model reduced to 300 dimensions.
- Original SGNS: The context vectors from a standard dependency-based Skip-Gram model.
- Reversed SGNS: A Skip-Gram model specifically trained to treat lemma-relation pairs as the prediction targets.
To analyze the geometry, they used t-SNE for visualization and Hierarchical Agglomerative Clustering (HAC) to find groupings of relations. Crucially, they employed within-lemma centering—subtracting a word's average vector from its relation-specific vectors—to isolate the geometric "fingerprint" of the syntactic relation itself, independent of the word's identity.
Datasets
- Training: WikiText-103 (approx. 103 million tokens from Wikipedia), parsed using the Stanza toolkit.
- Evaluation: The WordSim-353 benchmark (Similarity and Relatedness subsets) was used to test how well these embeddings align with human semantic judgments across 14 different dependency relations.
Key Insights and Ideas
- Context POS is the Global Architect: The most striking finding is that the global organization of these embeddings is driven by the Part-of-Speech (POS) of the context word, not the word being represented. For example, all relations that typically connect to a noun cluster together, regardless of whether the target word is a verb or an adjective.
- Syntactic Senses: Dependency relations act as natural filters for polysemy. A word like "bank" shifts its neighborhood from financial terms (insurance, mortgage) when used in a
compoundrelation to geographical terms (river, valley) when used in anobl(oblique) relation. - PPMI is More Consistent: Count-based models (PPMI) showed significantly higher geometric consistency across different relations than predictive models (SGNS). This suggests that SGNS might be more sensitive to the specific distributional quirks of individual relations.
- Context Vectors are Underrated: In SGNS models, the "context" vectors (often discarded after training) actually retained more benchmark-relevant semantic information for most relations than the specifically trained "target" vectors in the reversed configuration.
Key Results
- Global Alignment: There was a strong correlation ($\rho$ up to .603) between the distance of relation centroids and the differences in their context-POS distributions. In contrast, correlations with the target word's own POS were weakly negative.
- Local Neighborhoods: While broader lexical structures are preserved across relations, immediate "nearest neighbors" vary wildly. The chance-corrected Jaccard overlap for the top 10 neighbors across relations was low (under 0.15 for all models), highlighting how much the local semantic neighborhood changes with syntax.
- Cross-POS Differentiation: The models successfully separated different POS usages of the same word. For instance, "fine" as an adjective (
amod) clustered with "superb" and "exquisite," while "fine" as a noun (obj) clustered with "tax" and "fee."
Limitations and Future Work
The authors identify two main limitations:
- Descriptive Nature: The analysis of local geometric consistency is largely descriptive; the authors suggest that future work could benefit from more direct statistical significance testing of these structures.
- Manual Polysemy Analysis: The qualitative analysis of word senses relied on manually selected examples because current lexical similarity benchmarks aren't designed to evaluate sense distinctions tied specifically to syntactic relations.
Future research could explore how these structured semantic spaces can be leveraged for downstream tasks like Word Sense Disambiguation or improving the interpretability of large language models.
Summary updated